2024-03-18T20:04:21.166 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T20:04:21.166 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} --verbose --archive /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371 -- /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/orig.config.yaml 2024-03-18T20:04:21.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T20:04:21.324 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371 branch: wip-yuri10-testing-2024-03-15-1653-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} email: yweinste@redhat.com first_in_suite: false job_id: '7606371' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri10-testing-2024-03-15-1653-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-03-15-1653-reef sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8313 sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri10-testing-2024-03-15-1653-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIB8Ae+54B+tu+xrN4dy+6qA+Qfruw1AuNLlq7FRd96Jl8JDmjZYFUkn5qZsyajlFhIvrgjmUSUTkATTFEP91cE= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRDkwlMwjYLvjLrlv9nI3JnLihrK0ebfciUKdDCjNKRZYmxSYhZKqGJ4IFBpx2o1k4RzL5iGNDc94HW58lVrSI= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO4obnKNnSNvDcGwYZnClympqs9IHaPi5bfFR+G2hsP/xbY7M+yPYH8Cbot4/MPEyNc2pVWtu5Orl4lxv+NTig= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-16_15:03:17 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T20:04:21.324 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa; will attempt to use it 2024-03-18T20:04:21.325 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks 2024-03-18T20:04:21.325 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T20:04:21.327 INFO:teuthology.task.internal:Checking packages... 2024-03-18T20:04:21.350 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '49bdc51142759622a281d3e5327eebd07cbb8bb3' 2024-03-18T20:04:21.350 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T20:04:21.350 INFO:teuthology.packaging:ref: None 2024-03-18T20:04:21.350 INFO:teuthology.packaging:tag: None 2024-03-18T20:04:21.350 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-15-1653-reef 2024-03-18T20:04:21.350 INFO:teuthology.packaging:sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:04:21.350 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-03-15-1653-reef 2024-03-18T20:04:21.565 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-914.g49bdc511 2024-03-18T20:04:21.567 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T20:04:21.574 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T20:04:21.575 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T20:04:21.584 INFO:teuthology.task.internal:Saving configuration 2024-03-18T20:04:21.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T20:04:21.616 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:04:21.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.978053', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIB8Ae+54B+tu+xrN4dy+6qA+Qfruw1AuNLlq7FRd96Jl8JDmjZYFUkn5qZsyajlFhIvrgjmUSUTkATTFEP91cE='} 2024-03-18T20:04:21.662 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.976381', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRDkwlMwjYLvjLrlv9nI3JnLihrK0ebfciUKdDCjNKRZYmxSYhZKqGJ4IFBpx2o1k4RzL5iGNDc94HW58lVrSI='} 2024-03-18T20:04:21.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.974485', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO4obnKNnSNvDcGwYZnClympqs9IHaPi5bfFR+G2hsP/xbY7M+yPYH8Cbot4/MPEyNc2pVWtu5Orl4lxv+NTig='} 2024-03-18T20:04:21.685 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T20:04:21.693 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T20:04:21.693 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T20:04:21.693 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T20:04:21.693 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T20:04:21.800 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f47eba80160>, signals=[15]) 2024-03-18T20:04:21.801 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T20:04:21.809 INFO:teuthology.task.internal:Opening connections... 2024-03-18T20:04:21.809 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:04:21.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:04:21.900 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-18T20:04:21.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:04:21.973 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-18T20:04:21.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:04:22.048 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T20:04:22.056 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-18T20:04:22.076 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-18T20:04:22.076 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-18T20:04:22.135 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Red Hat Enterprise Linux" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8.6 (Ootpa)" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:ID="rhel" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="fedora" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8.6" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.redhat.com/" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:22.136 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-18T20:04:22.137 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-18T20:04:22.137 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-18T20:04:22.137 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-18T20:04:22.137 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-18T20:04:22.157 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-18T20:04:22.177 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-18T20:04:22.177 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-18T20:04:22.237 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2024-03-18T20:04:22.237 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2024-03-18T20:04:22.237 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-18T20:04:22.238 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-18T20:04:22.239 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-18T20:04:22.239 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-18T20:04:22.257 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-18T20:04:22.276 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-18T20:04:22.276 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Red Hat Enterprise Linux" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8.6 (Ootpa)" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:ID="rhel" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="fedora" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8.6" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-18T20:04:22.334 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.redhat.com/" 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-18T20:04:22.335 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-18T20:04:22.335 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-18T20:04:22.357 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T20:04:22.368 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T20:04:22.376 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T20:04:22.376 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-18T20:04:22.378 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-18T20:04:22.381 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-18T20:04:22.398 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T20:04:22.406 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T20:04:22.406 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-18T20:04:22.438 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-18T20:04:22.440 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-18T20:04:22.477 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T20:04:22.553 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T20:04:22.569 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-18T20:04:22.569 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-18T20:04:22.569 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T20:04:22.570 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T20:04:22.570 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T20:04:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:22.588 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-18T20:04:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:22.590 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-18T20:04:22.591 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-18T20:04:22.646 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.646 INFO:teuthology.task.kernel:Running kernel on smithi169: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-18T20:04:22.649 INFO:teuthology.orchestra.run.smithi136.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.650 INFO:teuthology.task.kernel:Running kernel on smithi136: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-18T20:04:22.651 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.651 INFO:teuthology.task.kernel:Running kernel on smithi098: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:22.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-18T20:04:24.087 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:24.087 INFO:teuthology.orchestra.run.smithi169.stdout:Unable to read consumer identity 2024-03-18T20:04:24.119 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:24.119 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2024-03-18T20:04:24.150 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:24.151 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-03-18T20:04:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:24.364 INFO:teuthology.orchestra.run.smithi098.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:24.601 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:24.601 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:24.602 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:24.628 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:24.628 INFO:teuthology.orchestra.run.smithi169.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:24.628 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:26.082 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-03-18T20:04:26.406 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-18T20:04:26.486 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-03-18T20:04:27.251 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-18T20:04:29.094 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 16 MB 00:02 2024-03-18T20:04:29.265 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 16 MB 00:02 2024-03-18T20:04:30.833 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 446 kB/s | 24 kB 00:00 2024-03-18T20:04:31.544 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-03-18T20:04:31.544 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-03-18T20:04:31.638 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-18T20:04:31.639 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-18T20:04:31.640 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-18T20:04:31.799 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2024-03-18T20:04:32.133 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:32.133 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2024-03-18T20:04:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:32.388 INFO:teuthology.orchestra.run.smithi098.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:32.691 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 319 kB/s | 24 kB 00:00 2024-03-18T20:04:32.697 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:02 ago on Mon 18 Mar 2024 08:04:30 PM UTC. 2024-03-18T20:04:32.765 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:32.766 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:32.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2024-03-18T20:04:32.882 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 326 kB/s | 24 kB 00:00 2024-03-18T20:04:33.179 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:33.179 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2024-03-18T20:04:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:33.414 INFO:teuthology.orchestra.run.smithi098.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:04:33.414 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-03-18T20:04:33.414 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-03-18T20:04:33.475 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-18T20:04:33.476 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-18T20:04:33.476 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-18T20:04:33.587 DEBUG:teuthology.orchestra.run.smithi136:> echo no | sudo yum reinstall kernel || true 2024-03-18T20:04:33.610 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-03-18T20:04:33.610 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-03-18T20:04:33.687 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-18T20:04:33.688 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-18T20:04:33.688 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-18T20:04:33.713 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Mar 2024 08:04:30 PM UTC. 2024-03-18T20:04:33.774 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:33.775 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:33.808 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:33.814 DEBUG:teuthology.orchestra.run.smithi169:> echo no | sudo yum reinstall kernel || true 2024-03-18T20:04:33.854 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T20:04:33.920 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:33.920 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-03-18T20:04:34.017 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:34.017 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:34.018 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-03-18T20:04:34.018 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-18T20:04:34.018 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T20:04:34.051 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-18T20:04:34.067 DEBUG:teuthology.parallel:result is None 2024-03-18T20:04:34.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:34.117 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:34.117 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:34.150 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:34.150 INFO:teuthology.orchestra.run.smithi169.stdout:Unable to read consumer identity 2024-03-18T20:04:34.378 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:34.378 INFO:teuthology.orchestra.run.smithi169.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:34.378 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:34.428 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:02 ago on Mon 18 Mar 2024 08:04:32 PM UTC. 2024-03-18T20:04:34.502 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:34.502 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:34.534 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:34.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum reinstall -y kernel || true 2024-03-18T20:04:34.695 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:02 ago on Mon 18 Mar 2024 08:04:32 PM UTC. 2024-03-18T20:04:34.765 INFO:teuthology.orchestra.run.smithi169.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:34.770 INFO:teuthology.orchestra.run.smithi169.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:34.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:34.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum reinstall -y kernel || true 2024-03-18T20:04:34.919 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:34.919 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-03-18T20:04:35.154 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:35.154 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:35.154 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:04:35.191 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:04:35.191 INFO:teuthology.orchestra.run.smithi169.stdout:Unable to read consumer identity 2024-03-18T20:04:35.390 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:35.390 INFO:teuthology.orchestra.run.smithi169.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-18T20:04:35.390 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:04:35.456 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Mar 2024 08:04:32 PM UTC. 2024-03-18T20:04:35.518 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:35.518 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:35.548 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:35.595 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T20:04:35.697 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:03 ago on Mon 18 Mar 2024 08:04:32 PM UTC. 2024-03-18T20:04:35.764 INFO:teuthology.orchestra.run.smithi169.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-03-18T20:04:35.764 INFO:teuthology.orchestra.run.smithi169.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-03-18T20:04:35.777 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:35.778 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:35.778 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-03-18T20:04:35.778 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-18T20:04:35.778 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T20:04:35.796 INFO:teuthology.orchestra.run.smithi169.stderr:Error: No packages marked for reinstall. 2024-03-18T20:04:35.810 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-18T20:04:35.823 DEBUG:teuthology.parallel:result is None 2024-03-18T20:04:35.843 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T20:04:35.997 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:35.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-03-18T20:04:35.997 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-03-18T20:04:35.997 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-18T20:04:35.997 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T20:04:36.031 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-18T20:04:36.046 DEBUG:teuthology.parallel:result is None 2024-03-18T20:04:36.046 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T20:04:36.056 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T20:04:36.056 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T20:04:36.059 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T20:04:36.061 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T20:04:36.104 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T20:04:36.112 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T20:04:36.125 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T20:04:36.125 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T20:04:36.127 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T20:04:36.129 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T20:04:36.177 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T20:04:36.184 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T20:04:36.185 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T20:04:36.194 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T20:04:36.197 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T20:04:36.222 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.232 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.247 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.257 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T20:04:36.259 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T20:04:36.266 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T20:04:36.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T20:04:36.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T20:04:36.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T20:04:36.325 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T20:04:36.335 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T20:04:36.335 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T20:04:36.353 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T20:04:36.355 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T20:04:36.382 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:36.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:36.509 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:36.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:36.636 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:04:36.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T20:04:36.703 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:36.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:36.799 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:36.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:36.924 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:04:36.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T20:04:36.990 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:37.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T20:04:37.095 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:37.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T20:04:37.222 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:04:37.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T20:04:37.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-18T20:04:37.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-18T20:04:37.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-18T20:04:37.337 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:04:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:04:37.380 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:04:37.752 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T20:04:37.761 INFO:teuthology.task.internal:Starting timer... 2024-03-18T20:04:37.761 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T20:04:37.781 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T20:04:37.795 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-18T20:04:37.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-18T20:04:37.850 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-18T20:04:37.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-18T20:04:37.921 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-18T20:04:37.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-18T20:04:37.991 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-18T20:04:37.993 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T20:04:37.993 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-18T20:04:38.030 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-18T20:04:38.030 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-18T20:04:38.051 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-18T20:04:38.051 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-18T20:04:38.070 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-18T20:04:38.070 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-03-18T20:04:38.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:04:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:38.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:04:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:38.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:04:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:04:38.152 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T20:04:38.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-18T20:04:38.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-18T20:04:38.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-18T20:04:38.233 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T20:04:38.244 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T20:04:38.249 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T20:04:38.373 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T20:04:38.388 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T20:04:38.390 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi136.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T20:14:26.839 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-03-18T20:14:26.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-18T20:14:26.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:14:26.921 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-18T20:14:26.987 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-18T20:14:26.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-18T20:14:26.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:14:27.061 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-18T20:14:27.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-18T20:14:27.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-18T20:14:27.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T20:14:27.209 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-18T20:14:27.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-18T20:14:27.281 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T20:14:27.295 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T20:14:27.296 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T20:14:27.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:14:27.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T20:14:27.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:14:27.301 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T20:14:27.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:14:27.329 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T20:14:27.334 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T20:14:27.343 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T20:14:27.350 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T20:14:27.355 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T20:14:27.372 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T20:14:27.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-18T20:14:27.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-18T20:14:27.424 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-18T20:14:27.427 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-18T20:14:27.438 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T20:14:27.442 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T20:14:27.449 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-18T20:14:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T20:14:27.457 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T20:14:27.461 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-18T20:14:27.476 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T20:14:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T20:14:27.536 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2024-03-18T20:14:27.539 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:14:27.539 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-18T20:14:27.539 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.539 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.540 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.540 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.540 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2024-03-18T20:14:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:14:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-18T20:14:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.543 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.544 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.583 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:14:27.585 INFO:teuthology.run_tasks:Running task pexec... 2024-03-18T20:14:27.594 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-18T20:14:27.594 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:14:27.595 DEBUG:teuthology.orchestra.run.smithi098:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-18T20:14:27.595 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi136.front.sepia.ceph.com 2024-03-18T20:14:27.595 DEBUG:teuthology.orchestra.run.smithi136:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-18T20:14:27.595 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi169.front.sepia.ceph.com 2024-03-18T20:14:27.596 DEBUG:teuthology.orchestra.run.smithi169:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-18T20:14:27.969 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:27.972 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:28.003 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:29.753 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-03-18T20:14:29.953 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-03-18T20:14:30.022 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-03-18T20:14:30.345 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 79 kB/s | 4.5 kB 00:00 2024-03-18T20:14:30.412 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-03-18T20:14:30.633 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-03-18T20:14:33.362 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-18T20:14:33.363 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-18T20:14:33.363 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-18T20:14:33.363 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-18T20:14:33.363 INFO:teuthology.orchestra.run.smithi136.stdout:Resetting modules: 2024-03-18T20:14:33.364 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools 2024-03-18T20:14:33.364 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:14:33.364 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-18T20:14:33.364 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-18T20:14:33.364 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:14:33.426 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-18T20:14:33.426 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-18T20:14:33.426 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout:Resetting modules: 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-18T20:14:33.427 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:14:33.444 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout:Resetting modules: 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:14:33.445 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-18T20:14:33.446 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-18T20:14:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:14:45.473 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-18T20:14:45.709 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-18T20:14:45.947 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:46.169 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:46.470 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-18T20:14:46.927 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:14:47.939 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-03-18T20:14:48.006 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-03-18T20:14:48.374 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 kB/s | 4.5 kB 00:00 2024-03-18T20:14:48.579 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2024-03-18T20:14:48.785 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-03-18T20:14:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-03-18T20:14:51.371 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-18T20:14:51.389 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-18T20:14:51.529 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================================== 2024-03-18T20:14:51.529 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-18T20:14:51.529 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================================== 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout:Installing group/module packages: 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-18T20:14:51.530 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:Installing module profiles: 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools/common 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools rhel8 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================================== 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-03-18T20:14:51.531 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:14:51.532 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 26 M 2024-03-18T20:14:51.532 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 87 M 2024-03-18T20:14:51.532 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-18T20:14:51.549 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================================== 2024-03-18T20:14:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================================== 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout:Installing group/module packages: 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-18T20:14:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:Installing module profiles: 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools/common 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling module streams: 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools rhel8 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================================== 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-03-18T20:14:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:14:51.552 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 26 M 2024-03-18T20:14:51.552 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 87 M 2024-03-18T20:14:51.552 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-18T20:14:51.669 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): criu-libs-3.18-4.module+el8.9.0+20326+3 287 kB/s | 39 kB 00:00 2024-03-18T20:14:51.685 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 711 kB/s | 94 kB 00:00 2024-03-18T20:14:51.701 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 166 kB/s | 25 kB 00:00 2024-03-18T20:14:51.719 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 133 kB/s | 25 kB 00:00 2024-03-18T20:14:51.727 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 226 kB/s | 39 kB 00:00 2024-03-18T20:14:51.745 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 441 kB/s | 94 kB 00:00 2024-03-18T20:14:51.777 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 445 kB/s | 41 kB 00:00 2024-03-18T20:14:51.812 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 288 kB/s | 41 kB 00:00 2024-03-18T20:14:51.854 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 7.5 MB/s | 1.0 MB 00:00 2024-03-18T20:14:51.872 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.9 MB/s | 1.0 MB 00:00 2024-03-18T20:14:51.920 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 5.1 MB/s | 739 kB 00:00 2024-03-18T20:14:51.953 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 2.9 MB/s | 239 kB 00:00 2024-03-18T20:14:51.962 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 2.2 MB/s | 239 kB 00:00 2024-03-18T20:14:52.021 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): cockpit-podman-75-1.module+el8.9.0+2124 3.5 MB/s | 739 kB 00:00 2024-03-18T20:14:52.104 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 23 MB/s | 8.8 MB 00:00 2024-03-18T20:14:52.137 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 837 kB/s | 154 kB 00:00 2024-03-18T20:14:52.146 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 22 MB/s | 8.8 MB 00:00 2024-03-18T20:14:52.171 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 1.0 MB/s | 154 kB 00:00 2024-03-18T20:14:52.221 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 12 MB/s | 3.7 MB 00:00 2024-03-18T20:14:52.282 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 12 MB/s | 3.7 MB 00:00 2024-03-18T20:14:52.354 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 12 MB/s | 2.5 MB 00:00 2024-03-18T20:14:52.363 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 13 MB/s | 2.5 MB 00:00 2024-03-18T20:14:52.380 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 312 kB/s | 49 kB 00:00 2024-03-18T20:14:52.408 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-18T20:14:52.413 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 378 kB/s | 49 kB 00:00 2024-03-18T20:14:52.488 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 21 MB/s | 8.1 MB 00:00 2024-03-18T20:14:52.489 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:14:52.489 INFO:teuthology.orchestra.run.smithi169.stdout:Total 27 MB/s | 26 MB 00:00 2024-03-18T20:14:52.514 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 22 MB/s | 8.1 MB 00:00 2024-03-18T20:14:52.514 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:14:52.514 INFO:teuthology.orchestra.run.smithi136.stdout:Total 26 MB/s | 26 MB 00:00 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================== 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================== 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout:Installing group/module packages: 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:14:52.571 INFO:teuthology.orchestra.run.smithi098.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout:Installing module profiles: 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools/common 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling module streams: 2024-03-18T20:14:52.572 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools rhel8 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================================== 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 26 M 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 87 M 2024-03-18T20:14:52.573 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-18T20:14:52.725 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 623 kB/s | 94 kB 00:00 2024-03-18T20:14:52.741 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 148 kB/s | 25 kB 00:00 2024-03-18T20:14:52.758 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 213 kB/s | 39 kB 00:00 2024-03-18T20:14:52.774 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-18T20:14:52.791 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-18T20:14:52.795 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-18T20:14:52.795 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-18T20:14:52.812 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-18T20:14:52.812 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-18T20:14:52.884 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 258 kB/s | 41 kB 00:00 2024-03-18T20:14:52.968 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 4.5 MB/s | 1.0 MB 00:00 2024-03-18T20:14:53.023 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-18T20:14:53.026 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 5.1 MB/s | 739 kB 00:00 2024-03-18T20:14:53.032 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-18T20:14:53.043 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-18T20:14:53.052 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-18T20:14:53.160 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): buildah-1.31.3-3.module+el8.9.0+21243+a 22 MB/s | 8.8 MB 00:00 2024-03-18T20:14:53.185 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.1 MB/s | 239 kB 00:00 2024-03-18T20:14:53.277 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): netavark-1.7.0-2.module+el8.9.0+21243+a 15 MB/s | 3.7 MB 00:00 2024-03-18T20:14:53.302 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-podman-4.6.0-2.module+el8.9.0+ 1.1 MB/s | 154 kB 00:00 2024-03-18T20:14:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 29 MB/s | 8.1 MB 00:00 2024-03-18T20:14:53.494 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 258 kB/s | 49 kB 00:00 2024-03-18T20:14:53.553 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 9.2 MB/s | 2.5 MB 00:00 2024-03-18T20:14:53.554 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:14:53.554 INFO:teuthology.orchestra.run.smithi098.stdout:Total 26 MB/s | 26 MB 00:00 2024-03-18T20:14:53.721 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-18T20:14:53.767 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-18T20:14:53.833 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-18T20:14:53.854 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-18T20:14:53.854 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-18T20:14:54.084 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-18T20:14:54.093 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-18T20:14:54.242 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-18T20:14:54.278 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-18T20:14:54.446 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-18T20:14:54.469 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-18T20:14:54.639 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-18T20:14:54.684 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-18T20:14:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-18T20:14:54.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-18T20:14:54.861 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-18T20:14:55.002 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-18T20:14:55.048 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-18T20:14:55.237 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-18T20:14:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-18T20:14:55.285 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-18T20:14:55.461 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-18T20:14:55.497 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-18T20:14:55.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-18T20:14:55.650 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-18T20:14:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-18T20:14:56.038 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-18T20:14:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-18T20:14:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:56.473 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-18T20:14:56.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:56.740 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:57.015 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-18T20:14:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-18T20:14:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:57.931 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-18T20:14:57.975 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-18T20:14:58.068 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-18T20:14:58.168 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-18T20:14:58.305 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-18T20:14:58.342 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:58.496 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:58.750 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-18T20:14:58.751 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-18T20:14:58.751 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-18T20:14:58.840 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-18T20:14:58.944 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:58.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-18T20:14:58.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-18T20:14:58.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-18T20:14:58.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-18T20:14:59.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-18T20:14:59.236 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-18T20:14:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-18T20:15:09.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-18T20:15:09.314 INFO:teuthology.orchestra.run.smithi169.stdout:Installed products updated. 2024-03-18T20:15:09.878 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:09.879 INFO:teuthology.orchestra.run.smithi169.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:09.880 INFO:teuthology.orchestra.run.smithi169.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-18T20:15:09.880 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:15:09.880 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-18T20:15:10.034 DEBUG:teuthology.parallel:result is None 2024-03-18T20:15:10.532 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-18T20:15:10.532 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-18T20:15:11.089 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.090 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:11.090 INFO:teuthology.orchestra.run.smithi136.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:11.090 INFO:teuthology.orchestra.run.smithi136.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-18T20:15:11.090 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:15:11.090 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-18T20:15:11.240 DEBUG:teuthology.parallel:result is None 2024-03-18T20:15:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-18T20:15:13.269 INFO:teuthology.orchestra.run.smithi098.stdout:Installed products updated. 2024-03-18T20:15:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-18T20:15:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-18T20:15:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-18T20:15:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:15:13.812 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-18T20:15:13.952 DEBUG:teuthology.parallel:result is None 2024-03-18T20:15:13.952 INFO:teuthology.run_tasks:Running task install... 2024-03-18T20:15:13.994 DEBUG:teuthology.task.install:project ceph 2024-03-18T20:15:13.994 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}} 2024-03-18T20:15:13.995 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'} 2024-03-18T20:15:13.995 INFO:teuthology.task.install:Using flavor: default 2024-03-18T20:15:14.004 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T20:15:14.004 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T20:15:14.004 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'tag': None, 'wait_for_package': False} 2024-03-18T20:15:14.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:15:14.007 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'tag': None, 'wait_for_package': False} 2024-03-18T20:15:14.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:15:14.009 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'tag': None, 'wait_for_package': False} 2024-03-18T20:15:14.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:15:14.151 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/ 2024-03-18T20:15:14.151 INFO:teuthology.task.install.rpm:Package version is 18.2.2-914.g49bdc511 2024-03-18T20:15:14.162 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/ 2024-03-18T20:15:14.162 INFO:teuthology.task.install.rpm:Package version is 18.2.2-914.g49bdc511 2024-03-18T20:15:14.171 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/ 2024-03-18T20:15:14.171 INFO:teuthology.task.install.rpm:Package version is 18.2.2-914.g49bdc511 2024-03-18T20:15:14.260 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T20:15:14.261 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:15:14.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T20:15:14.274 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T20:15:14.274 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:15:14.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T20:15:14.289 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T20:15:14.289 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49bdc51142759622a281d3e5327eebd07cbb8bb3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T20:15:14.307 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T20:15:14.307 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49bdc51142759622a281d3e5327eebd07cbb8bb3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T20:15:14.342 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T20:15:14.342 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:15:14.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T20:15:14.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T20:15:14.376 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T20:15:14.376 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49bdc51142759622a281d3e5327eebd07cbb8bb3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T20:15:14.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T20:15:14.435 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T20:15:14.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T20:15:14.462 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T20:15:14.501 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-18T20:15:14.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-18T20:15:14.529 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T20:15:14.531 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-03-18T20:15:14.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-03-18T20:15:14.597 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-18T20:15:14.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-18T20:15:14.881 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:14.913 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:14.979 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:16.160 INFO:teuthology.orchestra.run.smithi098.stdout:61 files removed 2024-03-18T20:15:16.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T20:15:16.334 INFO:teuthology.orchestra.run.smithi136.stdout:61 files removed 2024-03-18T20:15:16.342 INFO:teuthology.orchestra.run.smithi169.stdout:61 files removed 2024-03-18T20:15:16.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T20:15:16.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T20:15:16.554 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:16.722 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:16.731 INFO:teuthology.orchestra.run.smithi169.stdout:Updating Subscription Management repositories. 2024-03-18T20:15:18.576 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 177 kB/s | 80 kB 00:00 2024-03-18T20:15:18.616 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 162 kB/s | 80 kB 00:00 2024-03-18T20:15:18.859 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 122 kB/s | 80 kB 00:00 2024-03-18T20:15:19.033 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 26 kB/s | 9.9 kB 00:00 2024-03-18T20:15:19.202 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 17 kB/s | 9.9 kB 00:00 2024-03-18T20:15:19.570 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 15 kB/s | 9.9 kB 00:00 2024-03-18T20:15:19.645 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 3.6 kB/s | 1.5 kB 00:00 2024-03-18T20:15:19.667 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2024-03-18T20:15:19.996 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-03-18T20:15:20.002 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 3.7 kB/s | 1.5 kB 00:00 2024-03-18T20:15:20.043 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 9.9 kB/s | 3.5 kB 00:00 2024-03-18T20:15:20.302 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-03-18T20:15:21.155 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-18T20:15:24.713 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 317 kB/s | 24 kB 00:00 2024-03-18T20:15:25.593 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 2.9 MB/s | 16 MB 00:05 2024-03-18T20:15:25.998 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 59 MB 00:01 2024-03-18T20:15:29.377 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 89 kB/s | 24 kB 00:00 2024-03-18T20:15:30.778 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 66 MB 00:01 2024-03-18T20:15:37.134 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 66 MB 00:01 2024-03-18T20:15:38.412 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 915 kB/s | 16 MB 00:18 2024-03-18T20:15:40.894 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-03-18T20:15:42.043 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 299 kB/s | 24 kB 00:00 2024-03-18T20:15:43.386 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 59 MB 00:01 2024-03-18T20:15:43.920 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 59 MB 00:01 2024-03-18T20:15:47.576 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 12 MB/s | 8.8 MB 00:00 2024-03-18T20:15:54.173 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 15 MB/s | 8.8 MB 00:00 2024-03-18T20:15:54.528 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:15:54.529 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:15:54.547 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-18T20:15:54.548 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-18T20:15:54.550 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-18T20:15:54.551 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-18T20:15:54.554 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-18T20:15:54.558 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-18T20:15:54.559 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-18T20:15:54.784 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout:=================================================================================================================================== 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout:=================================================================================================================================== 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.4 k 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:18.2.2-914.g49bdc511.el8 ceph 5.4 M 2024-03-18T20:15:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 854 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-914.g49bdc511.el8 ceph 165 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:18.2.2-914.g49bdc511.el8 ceph 1.6 M 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 138 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 1.7 M 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 7.4 M 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 49 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 13 M 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:18.2.2-914.g49bdc511.el8 ceph 47 M 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 263 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 172 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 261 k 2024-03-18T20:15:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 18 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 31 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 775 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 139 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-18T20:15:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:18.2.2-914.g49bdc511.el8 ceph 196 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:18.2.2-914.g49bdc511.el8 ceph 370 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 365 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 107 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 95 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.5 M 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 182 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-18T20:15:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.7 M 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.6 M 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-18T20:15:54.802 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 22 M 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 23 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:18.2.2-914.g49bdc511.el8 ceph 2.3 M 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 245 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:18.2.2-914.g49bdc511.el8 ceph 4.6 M 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 18 M 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 15 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:18.2.2-914.g49bdc511.el8 ceph 24 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-18T20:15:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:18.2.2-914.g49bdc511.el8 ceph 142 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 474 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.5 M 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-18T20:15:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-18T20:15:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 45 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 126 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-18T20:15:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-18T20:15:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-18T20:15:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling module streams: 2024-03-18T20:15:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql 10 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:=================================================================================================================================== 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:Install 130 Packages 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 16 Packages 2024-03-18T20:15:54.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:15:54.811 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 212 M 2024-03-18T20:15:54.811 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-18T20:15:55.218 INFO:teuthology.orchestra.run.smithi098.stdout:(1/146): ceph-18.2.2-914.g49bdc511.el8.x86_64.r 19 kB/s | 6.4 kB 00:00 2024-03-18T20:15:56.026 INFO:teuthology.orchestra.run.smithi098.stdout:(2/146): ceph-fuse-18.2.2-914.g49bdc511.el8.x86 1.0 MB/s | 854 kB 00:00 2024-03-18T20:15:56.168 INFO:teuthology.orchestra.run.smithi098.stdout:(3/146): ceph-immutable-object-cache-18.2.2-914 1.1 MB/s | 165 kB 00:00 2024-03-18T20:15:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:(4/146): ceph-base-18.2.2-914.g49bdc511.el8.x86 3.1 MB/s | 5.4 MB 00:01 2024-03-18T20:15:56.869 INFO:teuthology.orchestra.run.smithi098.stdout:(5/146): ceph-mgr-18.2.2-914.g49bdc511.el8.x86_ 5.8 MB/s | 1.6 MB 00:00 2024-03-18T20:15:57.046 INFO:teuthology.orchestra.run.smithi169.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 66 MB 00:01 2024-03-18T20:15:57.227 INFO:teuthology.orchestra.run.smithi098.stdout:(6/146): ceph-mds-18.2.2-914.g49bdc511.el8.x86_ 2.2 MB/s | 2.3 MB 00:01 2024-03-18T20:15:57.528 INFO:teuthology.orchestra.run.smithi098.stdout:(7/146): ceph-mon-18.2.2-914.g49bdc511.el8.x86_ 7.0 MB/s | 4.6 MB 00:00 2024-03-18T20:15:58.395 INFO:teuthology.orchestra.run.smithi098.stdout:(8/146): ceph-common-18.2.2-914.g49bdc511.el8.x 6.2 MB/s | 22 MB 00:03 2024-03-18T20:15:58.445 INFO:teuthology.orchestra.run.smithi098.stdout:(9/146): ceph-selinux-18.2.2-914.g49bdc511.el8. 487 kB/s | 24 kB 00:00 2024-03-18T20:15:59.095 INFO:teuthology.orchestra.run.smithi098.stdout:(10/146): ceph-radosgw-18.2.2-914.g49bdc511.el8 8.4 MB/s | 13 MB 00:01 2024-03-18T20:15:59.170 INFO:teuthology.orchestra.run.smithi098.stdout:(11/146): cephfs-mirror-18.2.2-914.g49bdc511.el 3.4 MB/s | 261 kB 00:00 2024-03-18T20:15:59.229 INFO:teuthology.orchestra.run.smithi098.stdout:(12/146): libcephfs-devel-18.2.2-914.g49bdc511. 529 kB/s | 31 kB 00:00 2024-03-18T20:15:59.295 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:15:59.296 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:15:59.315 INFO:teuthology.orchestra.run.smithi136.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-18T20:15:59.316 INFO:teuthology.orchestra.run.smithi136.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-18T20:15:59.317 INFO:teuthology.orchestra.run.smithi136.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-18T20:15:59.319 INFO:teuthology.orchestra.run.smithi136.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-18T20:15:59.321 INFO:teuthology.orchestra.run.smithi136.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-18T20:15:59.325 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-18T20:15:59.327 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-18T20:15:59.346 INFO:teuthology.orchestra.run.smithi098.stdout:(13/146): libcephfs2-18.2.2-914.g49bdc511.el8.x 6.5 MB/s | 775 kB 00:00 2024-03-18T20:15:59.413 INFO:teuthology.orchestra.run.smithi098.stdout:(14/146): libcephsqlite-18.2.2-914.g49bdc511.el 2.1 MB/s | 142 kB 00:00 2024-03-18T20:15:59.480 INFO:teuthology.orchestra.run.smithi098.stdout:(15/146): librados-devel-18.2.2-914.g49bdc511.e 2.0 MB/s | 139 kB 00:00 2024-03-18T20:15:59.543 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-18T20:15:59.555 INFO:teuthology.orchestra.run.smithi098.stdout:(16/146): libradosstriper1-18.2.2-914.g49bdc511 6.2 MB/s | 474 kB 00:00 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout:=================================================================================================================================== 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout:=================================================================================================================================== 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.4 k 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:18.2.2-914.g49bdc511.el8 ceph 5.4 M 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 854 k 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-914.g49bdc511.el8 ceph 165 k 2024-03-18T20:15:59.557 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:18.2.2-914.g49bdc511.el8 ceph 1.6 M 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 138 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 1.7 M 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 7.4 M 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 49 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 13 M 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:18.2.2-914.g49bdc511.el8 ceph 47 M 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 263 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 172 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 261 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 18 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-18T20:15:59.558 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 31 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 775 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 139 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:18.2.2-914.g49bdc511.el8 ceph 196 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:18.2.2-914.g49bdc511.el8 ceph 370 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 365 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 107 k 2024-03-18T20:15:59.559 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 95 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.5 M 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 182 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.7 M 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.6 M 2024-03-18T20:15:59.560 INFO:teuthology.orchestra.run.smithi136.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 22 M 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 23 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:18.2.2-914.g49bdc511.el8 ceph 2.3 M 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 245 k 2024-03-18T20:15:59.561 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:18.2.2-914.g49bdc511.el8 ceph 4.6 M 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 18 M 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 15 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:18.2.2-914.g49bdc511.el8 ceph 24 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:18.2.2-914.g49bdc511.el8 ceph 142 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-18T20:15:59.562 INFO:teuthology.orchestra.run.smithi136.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 474 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.5 M 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-18T20:15:59.563 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 45 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 126 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-18T20:15:59.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-18T20:15:59.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-18T20:15:59.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-18T20:15:59.567 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql 10 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout:=================================================================================================================================== 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout:Install 130 Packages 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 16 Packages 2024-03-18T20:15:59.568 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:15:59.569 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 212 M 2024-03-18T20:15:59.569 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-18T20:16:00.114 INFO:teuthology.orchestra.run.smithi136.stdout:(1/146): ceph-18.2.2-914.g49bdc511.el8.x86_64.r 13 kB/s | 6.4 kB 00:00 2024-03-18T20:16:00.247 INFO:teuthology.orchestra.run.smithi098.stdout:(17/146): librgw2-18.2.2-914.g49bdc511.el8.x86_ 9.5 MB/s | 6.5 MB 00:00 2024-03-18T20:16:00.305 INFO:teuthology.orchestra.run.smithi098.stdout:(18/146): python3-ceph-argparse-18.2.2-914.g49b 767 kB/s | 45 kB 00:00 2024-03-18T20:16:00.373 INFO:teuthology.orchestra.run.smithi098.stdout:(19/146): python3-ceph-common-18.2.2-914.g49bdc 1.8 MB/s | 126 kB 00:00 2024-03-18T20:16:00.448 INFO:teuthology.orchestra.run.smithi098.stdout:(20/146): python3-cephfs-18.2.2-914.g49bdc511.e 2.6 MB/s | 196 kB 00:00 2024-03-18T20:16:00.523 INFO:teuthology.orchestra.run.smithi098.stdout:(21/146): python3-rados-18.2.2-914.g49bdc511.el 4.8 MB/s | 370 kB 00:00 2024-03-18T20:16:00.599 INFO:teuthology.orchestra.run.smithi098.stdout:(22/146): python3-rbd-18.2.2-914.g49bdc511.el8. 4.7 MB/s | 365 kB 00:00 2024-03-18T20:16:00.666 INFO:teuthology.orchestra.run.smithi098.stdout:(23/146): python3-rgw-18.2.2-914.g49bdc511.el8. 1.6 MB/s | 107 kB 00:00 2024-03-18T20:16:00.736 INFO:teuthology.orchestra.run.smithi136.stdout:(2/146): ceph-fuse-18.2.2-914.g49bdc511.el8.x86 1.3 MB/s | 854 kB 00:00 2024-03-18T20:16:00.749 INFO:teuthology.orchestra.run.smithi098.stdout:(24/146): rbd-fuse-18.2.2-914.g49bdc511.el8.x86 1.1 MB/s | 95 kB 00:00 2024-03-18T20:16:00.886 INFO:teuthology.orchestra.run.smithi136.stdout:(3/146): ceph-immutable-object-cache-18.2.2-914 1.1 MB/s | 165 kB 00:00 2024-03-18T20:16:01.250 INFO:teuthology.orchestra.run.smithi098.stdout:(25/146): rbd-mirror-18.2.2-914.g49bdc511.el8.x 6.9 MB/s | 3.5 MB 00:00 2024-03-18T20:16:01.341 INFO:teuthology.orchestra.run.smithi098.stdout:(26/146): rbd-nbd-18.2.2-914.g49bdc511.el8.x86_ 1.9 MB/s | 182 kB 00:00 2024-03-18T20:16:01.442 INFO:teuthology.orchestra.run.smithi098.stdout:(27/146): ceph-grafana-dashboards-18.2.2-914.g4 233 kB/s | 23 kB 00:00 2024-03-18T20:16:01.845 INFO:teuthology.orchestra.run.smithi136.stdout:(4/146): ceph-mds-18.2.2-914.g49bdc511.el8.x86_ 2.4 MB/s | 2.3 MB 00:00 2024-03-18T20:16:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:(28/146): ceph-test-18.2.2-914.g49bdc511.el8.x8 14 MB/s | 47 MB 00:03 2024-03-18T20:16:01.909 INFO:teuthology.orchestra.run.smithi098.stdout:(29/146): ceph-mgr-cephadm-18.2.2-914.g49bdc511 295 kB/s | 138 kB 00:00 2024-03-18T20:16:01.945 INFO:teuthology.orchestra.run.smithi136.stdout:(5/146): ceph-base-18.2.2-914.g49bdc511.el8.x86 2.3 MB/s | 5.4 MB 00:02 2024-03-18T20:16:02.146 INFO:teuthology.orchestra.run.smithi098.stdout:(30/146): ceph-mgr-dashboard-18.2.2-914.g49bdc5 6.7 MB/s | 1.7 MB 00:00 2024-03-18T20:16:02.210 INFO:teuthology.orchestra.run.smithi098.stdout:(31/146): ceph-mgr-modules-core-18.2.2-914.g49b 3.8 MB/s | 245 kB 00:00 2024-03-18T20:16:02.268 INFO:teuthology.orchestra.run.smithi098.stdout:(32/146): ceph-mgr-rook-18.2.2-914.g49bdc511.el 844 kB/s | 49 kB 00:00 2024-03-18T20:16:02.287 INFO:teuthology.orchestra.run.smithi136.stdout:(6/146): ceph-mgr-18.2.2-914.g49bdc511.el8.x86_ 3.5 MB/s | 1.6 MB 00:00 2024-03-18T20:16:02.327 INFO:teuthology.orchestra.run.smithi098.stdout:(33/146): ceph-prometheus-alerts-18.2.2-914.g49 253 kB/s | 15 kB 00:00 2024-03-18T20:16:02.394 INFO:teuthology.orchestra.run.smithi098.stdout:(34/146): ceph-volume-18.2.2-914.g49bdc511.el8. 3.9 MB/s | 263 kB 00:00 2024-03-18T20:16:02.461 INFO:teuthology.orchestra.run.smithi098.stdout:(35/146): cephadm-18.2.2-914.g49bdc511.el8.noar 2.5 MB/s | 172 kB 00:00 2024-03-18T20:16:02.528 INFO:teuthology.orchestra.run.smithi098.stdout:(36/146): cephfs-top-18.2.2-914.g49bdc511.el8.n 272 kB/s | 18 kB 00:00 2024-03-18T20:16:03.003 INFO:teuthology.orchestra.run.smithi098.stdout:(37/146): ceph-mgr-diskprediction-local-18.2.2- 6.8 MB/s | 7.4 MB 00:01 2024-03-18T20:16:03.053 INFO:teuthology.orchestra.run.smithi098.stdout:(38/146): gperftools-libs-2.7-9.el8.x86_64.rpm 583 kB/s | 306 kB 00:00 2024-03-18T20:16:03.162 INFO:teuthology.orchestra.run.smithi098.stdout:(39/146): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2024-03-18T20:16:03.320 INFO:teuthology.orchestra.run.smithi098.stdout:(40/146): liboath-2.6.2-3.el8.x86_64.rpm 185 kB/s | 59 kB 00:00 2024-03-18T20:16:03.363 INFO:teuthology.orchestra.run.smithi098.stdout:(41/146): python3-asyncssh-2.7.0-2.el8.noarch.r 2.3 MB/s | 476 kB 00:00 2024-03-18T20:16:03.404 INFO:teuthology.orchestra.run.smithi098.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 523 kB/s | 44 kB 00:00 2024-03-18T20:16:03.455 INFO:teuthology.orchestra.run.smithi098.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2024-03-18T20:16:03.497 INFO:teuthology.orchestra.run.smithi098.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 358 kB/s | 33 kB 00:00 2024-03-18T20:16:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 177 kB/s | 16 kB 00:00 2024-03-18T20:16:03.628 INFO:teuthology.orchestra.run.smithi136.stdout:(7/146): ceph-common-18.2.2-914.g49bdc511.el8.x 5.4 MB/s | 22 MB 00:04 2024-03-18T20:16:03.764 INFO:teuthology.orchestra.run.smithi098.stdout:(46/146): ceph-osd-18.2.2-914.g49bdc511.el8.x86 2.7 MB/s | 18 MB 00:06 2024-03-18T20:16:03.790 INFO:teuthology.orchestra.run.smithi098.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 590 kB/s | 173 kB 00:00 2024-03-18T20:16:03.815 INFO:teuthology.orchestra.run.smithi098.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 1.4 MB/s | 384 kB 00:00 2024-03-18T20:16:03.882 INFO:teuthology.orchestra.run.smithi098.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-03-18T20:16:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 70 kB/s | 18 kB 00:00 2024-03-18T20:16:04.221 INFO:teuthology.orchestra.run.smithi136.stdout:(8/146): ceph-osd-18.2.2-914.g49bdc511.el8.x86_ 9.2 MB/s | 18 MB 00:01 2024-03-18T20:16:04.241 INFO:teuthology.orchestra.run.smithi098.stdout:(51/146): python3-google-auth-1.1.1-10.el8.noar 171 kB/s | 81 kB 00:00 2024-03-18T20:16:04.266 INFO:teuthology.orchestra.run.smithi098.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 243 kB/s | 49 kB 00:00 2024-03-18T20:16:04.321 INFO:teuthology.orchestra.run.smithi136.stdout:(9/146): ceph-selinux-18.2.2-914.g49bdc511.el8. 243 kB/s | 24 kB 00:00 2024-03-18T20:16:04.349 INFO:teuthology.orchestra.run.smithi098.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 542 kB/s | 59 kB 00:00 2024-03-18T20:16:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 554 kB/s | 60 kB 00:00 2024-03-18T20:16:04.454 INFO:teuthology.orchestra.run.smithi136.stdout:(10/146): ceph-mon-18.2.2-914.g49bdc511.el8.x86 1.8 MB/s | 4.6 MB 00:02 2024-03-18T20:16:04.483 INFO:teuthology.orchestra.run.smithi098.stdout:(55/146): python3-kubernetes-11.0.0-6.el8.noarc 1.9 MB/s | 1.2 MB 00:00 2024-03-18T20:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 117 kB/s | 16 kB 00:00 2024-03-18T20:16:04.609 INFO:teuthology.orchestra.run.smithi098.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 269 kB/s | 34 kB 00:00 2024-03-18T20:16:04.634 INFO:teuthology.orchestra.run.smithi098.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2024-03-18T20:16:04.663 INFO:teuthology.orchestra.run.smithi136.stdout:(11/146): cephfs-mirror-18.2.2-914.g49bdc511.el 1.2 MB/s | 261 kB 00:00 2024-03-18T20:16:04.693 INFO:teuthology.orchestra.run.smithi098.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 768 kB/s | 64 kB 00:00 2024-03-18T20:16:04.726 INFO:teuthology.orchestra.run.smithi098.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2024-03-18T20:16:04.752 INFO:teuthology.orchestra.run.smithi098.stdout:(61/146): python3-pecan-1.3.2-9.el8.noarch.rpm 703 kB/s | 283 kB 00:00 2024-03-18T20:16:04.771 INFO:teuthology.orchestra.run.smithi136.stdout:(12/146): libcephfs-devel-18.2.2-914.g49bdc511. 285 kB/s | 31 kB 00:00 2024-03-18T20:16:04.785 INFO:teuthology.orchestra.run.smithi098.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2024-03-18T20:16:04.819 INFO:teuthology.orchestra.run.smithi098.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 317 kB/s | 29 kB 00:00 2024-03-18T20:16:04.844 INFO:teuthology.orchestra.run.smithi098.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 298 kB/s | 27 kB 00:00 2024-03-18T20:16:04.886 INFO:teuthology.orchestra.run.smithi098.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 2.5 MB/s | 255 kB 00:00 2024-03-18T20:16:04.920 INFO:teuthology.orchestra.run.smithi098.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2024-03-18T20:16:04.945 INFO:teuthology.orchestra.run.smithi098.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 609 kB/s | 61 kB 00:00 2024-03-18T20:16:04.979 INFO:teuthology.orchestra.run.smithi098.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 926 kB/s | 85 kB 00:00 2024-03-18T20:16:05.021 INFO:teuthology.orchestra.run.smithi136.stdout:(13/146): ceph-radosgw-18.2.2-914.g49bdc511.el8 9.4 MB/s | 13 MB 00:01 2024-03-18T20:16:05.037 INFO:teuthology.orchestra.run.smithi098.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 70 kB 00:00 2024-03-18T20:16:05.063 INFO:teuthology.orchestra.run.smithi098.stdout:(70/146): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2024-03-18T20:16:05.113 INFO:teuthology.orchestra.run.smithi136.stdout:(14/146): libcephsqlite-18.2.2-914.g49bdc511.el 1.5 MB/s | 142 kB 00:00 2024-03-18T20:16:05.147 INFO:teuthology.orchestra.run.smithi136.stdout:(15/146): libcephfs2-18.2.2-914.g49bdc511.el8.x 2.0 MB/s | 775 kB 00:00 2024-03-18T20:16:05.196 INFO:teuthology.orchestra.run.smithi098.stdout:(71/146): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2024-03-18T20:16:05.214 INFO:teuthology.orchestra.run.smithi136.stdout:(16/146): librados-devel-18.2.2-914.g49bdc511.e 1.4 MB/s | 139 kB 00:00 2024-03-18T20:16:05.322 INFO:teuthology.orchestra.run.smithi098.stdout:(72/146): python3-prettytable-0.7.2-14.el8.noar 170 kB/s | 44 kB 00:00 2024-03-18T20:16:05.347 INFO:teuthology.orchestra.run.smithi098.stdout:(73/146): python3-pyasn1-modules-0.3.7-6.el8.no 732 kB/s | 110 kB 00:00 2024-03-18T20:16:05.447 INFO:teuthology.orchestra.run.smithi098.stdout:(74/146): python3-jsonpatch-1.21-2.el8.noarch.r 212 kB/s | 27 kB 00:00 2024-03-18T20:16:05.481 INFO:teuthology.orchestra.run.smithi098.stdout:(75/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 945 kB/s | 126 kB 00:00 2024-03-18T20:16:05.581 INFO:teuthology.orchestra.run.smithi098.stdout:(76/146): python3-werkzeug-0.12.2-4.el8.noarch. 3.3 MB/s | 457 kB 00:00 2024-03-18T20:16:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:(77/146): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2024-03-18T20:16:05.732 INFO:teuthology.orchestra.run.smithi098.stdout:(78/146): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2024-03-18T20:16:05.739 INFO:teuthology.orchestra.run.smithi136.stdout:(17/146): libradosstriper1-18.2.2-914.g49bdc511 801 kB/s | 474 kB 00:00 2024-03-18T20:16:05.766 INFO:teuthology.orchestra.run.smithi098.stdout:(79/146): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2024-03-18T20:16:05.831 INFO:teuthology.orchestra.run.smithi136.stdout:(18/146): python3-ceph-argparse-18.2.2-914.g49b 488 kB/s | 45 kB 00:00 2024-03-18T20:16:05.866 INFO:teuthology.orchestra.run.smithi098.stdout:(80/146): python3-markupsafe-0.23-19.el8.x86_64 290 kB/s | 39 kB 00:00 2024-03-18T20:16:05.915 INFO:teuthology.orchestra.run.smithi136.stdout:(19/146): python3-ceph-common-18.2.2-914.g49bdc 1.5 MB/s | 126 kB 00:00 2024-03-18T20:16:05.916 INFO:teuthology.orchestra.run.smithi098.stdout:(81/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2024-03-18T20:16:05.942 INFO:teuthology.orchestra.run.smithi098.stdout:(82/146): uuid-1.6.2-43.el8.x86_64.rpm 851 kB/s | 64 kB 00:00 2024-03-18T20:16:06.025 INFO:teuthology.orchestra.run.smithi098.stdout:(83/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 954 kB/s | 103 kB 00:00 2024-03-18T20:16:06.059 INFO:teuthology.orchestra.run.smithi098.stdout:(84/146): thrift-0.13.0-2.el8.x86_64.rpm 1.6 MB/s | 1.7 MB 00:01 2024-03-18T20:16:06.073 INFO:teuthology.orchestra.run.smithi136.stdout:(20/146): python3-cephfs-18.2.2-914.g49bdc511.e 1.2 MB/s | 196 kB 00:00 2024-03-18T20:16:06.148 INFO:teuthology.orchestra.run.smithi136.stdout:(21/146): librgw2-18.2.2-914.g49bdc511.el8.x86_ 7.0 MB/s | 6.5 MB 00:00 2024-03-18T20:16:06.184 INFO:teuthology.orchestra.run.smithi098.stdout:(85/146): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2024-03-18T20:16:06.217 INFO:teuthology.orchestra.run.smithi098.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 2.7 MB/s | 538 kB 00:00 2024-03-18T20:16:06.270 INFO:teuthology.orchestra.run.smithi136.stdout:(22/146): python3-rbd-18.2.2-914.g49bdc511.el8. 2.9 MB/s | 365 kB 00:00 2024-03-18T20:16:06.299 INFO:teuthology.orchestra.run.smithi136.stdout:(23/146): python3-rados-18.2.2-914.g49bdc511.el 1.6 MB/s | 370 kB 00:00 2024-03-18T20:16:06.335 INFO:teuthology.orchestra.run.smithi098.stdout:(87/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-03-18T20:16:06.360 INFO:teuthology.orchestra.run.smithi098.stdout:(88/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2024-03-18T20:16:06.366 INFO:teuthology.orchestra.run.smithi136.stdout:(24/146): python3-rgw-18.2.2-914.g49bdc511.el8. 1.1 MB/s | 107 kB 00:00 2024-03-18T20:16:06.391 INFO:teuthology.orchestra.run.smithi136.stdout:(25/146): rbd-fuse-18.2.2-914.g49bdc511.el8.x86 1.0 MB/s | 95 kB 00:00 2024-03-18T20:16:06.461 INFO:teuthology.orchestra.run.smithi098.stdout:(89/146): python3-numpy-1.14.3-10.el8.x86_64.rp 9.3 MB/s | 3.7 MB 00:00 2024-03-18T20:16:06.485 INFO:teuthology.orchestra.run.smithi098.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2024-03-18T20:16:06.575 INFO:teuthology.orchestra.run.smithi136.stdout:(26/146): rbd-nbd-18.2.2-914.g49bdc511.el8.x86_ 992 kB/s | 182 kB 00:00 2024-03-18T20:16:06.619 INFO:teuthology.orchestra.run.smithi098.stdout:(91/146): openblas-threads-0.3.15-6.el8.x86_64. 20 MB/s | 5.2 MB 00:00 2024-03-18T20:16:06.645 INFO:teuthology.orchestra.run.smithi098.stdout:(92/146): python3-mako-1.0.6-14.el8.noarch.rpm 854 kB/s | 157 kB 00:00 2024-03-18T20:16:06.761 INFO:teuthology.orchestra.run.smithi098.stdout:(93/146): openblas-0.3.15-6.el8.x86_64.rpm 18 MB/s | 5.0 MB 00:00 2024-03-18T20:16:06.803 INFO:teuthology.orchestra.run.smithi098.stdout:(94/146): automake-1.16.1-8.el8.noarch.rpm 3.8 MB/s | 714 kB 00:00 2024-03-18T20:16:06.828 INFO:teuthology.orchestra.run.smithi098.stdout:(95/146): libpq-13.11-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2024-03-18T20:16:06.854 INFO:teuthology.orchestra.run.smithi098.stdout:(96/146): jq-1.6-7.el8.x86_64.rpm 2.1 MB/s | 202 kB 00:00 2024-03-18T20:16:06.875 INFO:teuthology.orchestra.run.smithi136.stdout:(27/146): rbd-mirror-18.2.2-914.g49bdc511.el8.x 6.8 MB/s | 3.5 MB 00:00 2024-03-18T20:16:06.900 INFO:teuthology.orchestra.run.smithi136.stdout:(28/146): ceph-grafana-dashboards-18.2.2-914.g4 71 kB/s | 23 kB 00:00 2024-03-18T20:16:06.904 INFO:teuthology.orchestra.run.smithi098.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 540 kB/s | 54 kB 00:00 2024-03-18T20:16:06.996 INFO:teuthology.orchestra.run.smithi098.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-03-18T20:16:07.221 INFO:teuthology.orchestra.run.smithi098.stdout:(99/146): python3-scipy-1.0.0-21.module+el8.9.0 35 MB/s | 14 MB 00:00 2024-03-18T20:16:07.255 INFO:teuthology.orchestra.run.smithi098.stdout:(100/146): postgresql-contrib-10.23-4.module+el 3.1 MB/s | 812 kB 00:00 2024-03-18T20:16:07.297 INFO:teuthology.orchestra.run.smithi098.stdout:(101/146): postgresql-10.23-4.module+el8.9.0+21 3.8 MB/s | 1.5 MB 00:00 2024-03-18T20:16:07.355 INFO:teuthology.orchestra.run.smithi098.stdout:(102/146): libverto-devel-0.3.0-5.el8.x86_64.rp 178 kB/s | 18 kB 00:00 2024-03-18T20:16:07.367 INFO:teuthology.orchestra.run.smithi136.stdout:(29/146): ceph-test-18.2.2-914.g49bdc511.el8.x8 15 MB/s | 47 MB 00:03 2024-03-18T20:16:07.393 INFO:teuthology.orchestra.run.smithi136.stdout:(30/146): ceph-mgr-cephadm-18.2.2-914.g49bdc511 266 kB/s | 138 kB 00:00 2024-03-18T20:16:07.473 INFO:teuthology.orchestra.run.smithi098.stdout:(103/146): postgresql-server-10.23-4.module+el8 20 MB/s | 5.1 MB 00:00 2024-03-18T20:16:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:(104/146): python3-jwt-1.6.1-2.el8.noarch.rpm 212 kB/s | 43 kB 00:00 2024-03-18T20:16:07.523 INFO:teuthology.orchestra.run.smithi098.stdout:(105/146): python3-oauthlib-2.1.0-1.el8.noarch. 925 kB/s | 155 kB 00:00 2024-03-18T20:16:07.560 INFO:teuthology.orchestra.run.smithi136.stdout:(31/146): ceph-mgr-modules-core-18.2.2-914.g49b 1.4 MB/s | 245 kB 00:00 2024-03-18T20:16:07.624 INFO:teuthology.orchestra.run.smithi098.stdout:(106/146): python3-pycparser-2.14-14.el8.noarch 725 kB/s | 109 kB 00:00 2024-03-18T20:16:07.649 INFO:teuthology.orchestra.run.smithi098.stdout:(107/146): libaio-devel-0.3.112-1.el8.x86_64.rp 149 kB/s | 19 kB 00:00 2024-03-18T20:16:07.674 INFO:teuthology.orchestra.run.smithi098.stdout:(108/146): python3-requests-oauthlib-1.0.0-1.el 243 kB/s | 43 kB 00:00 2024-03-18T20:16:07.716 INFO:teuthology.orchestra.run.smithi098.stdout:(109/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 475 kB/s | 44 kB 00:00 2024-03-18T20:16:07.775 INFO:teuthology.orchestra.run.smithi098.stdout:(110/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2024-03-18T20:16:07.835 INFO:teuthology.orchestra.run.smithi136.stdout:(32/146): ceph-mgr-dashboard-18.2.2-914.g49bdc5 1.9 MB/s | 1.7 MB 00:00 2024-03-18T20:16:07.839 INFO:teuthology.orchestra.run.smithi098.stdout:(111/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.1 MB/s | 605 kB 00:00 2024-03-18T20:16:07.867 INFO:teuthology.orchestra.run.smithi098.stdout:(112/146): libselinux-devel-2.9-5.el8.x86_64.rp 2.1 MB/s | 200 kB 00:00 2024-03-18T20:16:07.877 INFO:teuthology.orchestra.run.smithi136.stdout:(33/146): ceph-mgr-rook-18.2.2-914.g49bdc511.el 155 kB/s | 49 kB 00:00 2024-03-18T20:16:07.884 INFO:teuthology.orchestra.run.smithi098.stdout:(113/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-03-18T20:16:07.902 INFO:teuthology.orchestra.run.smithi136.stdout:(34/146): ceph-prometheus-alerts-18.2.2-914.g49 222 kB/s | 15 kB 00:00 2024-03-18T20:16:07.942 INFO:teuthology.orchestra.run.smithi098.stdout:(114/146): libsepol-devel-2.9-3.el8.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-03-18T20:16:07.968 INFO:teuthology.orchestra.run.smithi098.stdout:(115/146): keyutils-libs-devel-1.5.10-9.el8.x86 376 kB/s | 48 kB 00:00 2024-03-18T20:16:08.011 INFO:teuthology.orchestra.run.smithi136.stdout:(35/146): ceph-mgr-diskprediction-local-18.2.2- 12 MB/s | 7.4 MB 00:00 2024-03-18T20:16:08.035 INFO:teuthology.orchestra.run.smithi098.stdout:(116/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 383 kB/s | 58 kB 00:00 2024-03-18T20:16:08.036 INFO:teuthology.orchestra.run.smithi136.stdout:(36/146): cephadm-18.2.2-914.g49bdc511.el8.noar 1.3 MB/s | 172 kB 00:00 2024-03-18T20:16:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 329 kB/s | 39 kB 00:00 2024-03-18T20:16:08.061 INFO:teuthology.orchestra.run.smithi136.stdout:(37/146): ceph-volume-18.2.2-914.g49bdc511.el8. 1.4 MB/s | 263 kB 00:00 2024-03-18T20:16:08.110 INFO:teuthology.orchestra.run.smithi098.stdout:(118/146): fuse-2.9.7-17.el8.x86_64.rpm 583 kB/s | 83 kB 00:00 2024-03-18T20:16:08.130 INFO:teuthology.orchestra.run.smithi136.stdout:(38/146): cephfs-top-18.2.2-914.g49bdc511.el8.n 152 kB/s | 18 kB 00:00 2024-03-18T20:16:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:(119/146): libgfortran-8.5.0-20.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2024-03-18T20:16:08.269 INFO:teuthology.orchestra.run.smithi098.stdout:(120/146): libuuid-devel-2.32.1-43.el8.x86_64.r 628 kB/s | 100 kB 00:00 2024-03-18T20:16:08.303 INFO:teuthology.orchestra.run.smithi098.stdout:(121/146): libquadmath-8.5.0-20.el8.x86_64.rpm 711 kB/s | 172 kB 00:00 2024-03-18T20:16:08.337 INFO:teuthology.orchestra.run.smithi136.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 214 kB/s | 59 kB 00:00 2024-03-18T20:16:08.362 INFO:teuthology.orchestra.run.smithi136.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 324 kB/s | 75 kB 00:00 2024-03-18T20:16:08.437 INFO:teuthology.orchestra.run.smithi136.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 582 kB/s | 44 kB 00:00 2024-03-18T20:16:08.445 INFO:teuthology.orchestra.run.smithi098.stdout:(122/146): NetworkManager-initscripts-updown-1. 1.0 MB/s | 144 kB 00:00 2024-03-18T20:16:08.470 INFO:teuthology.orchestra.run.smithi098.stdout:(123/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 238 kB 00:00 2024-03-18T20:16:08.495 INFO:teuthology.orchestra.run.smithi098.stdout:(124/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 321 kB/s | 73 kB 00:00 2024-03-18T20:16:08.529 INFO:teuthology.orchestra.run.smithi136.stdout:(42/146): gperftools-libs-2.7-9.el8.x86_64.rpm 621 kB/s | 306 kB 00:00 2024-03-18T20:16:08.588 INFO:teuthology.orchestra.run.smithi136.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 567 kB/s | 33 kB 00:00 2024-03-18T20:16:08.612 INFO:teuthology.orchestra.run.smithi098.stdout:(125/146): elfutils-libelf-devel-0.189-3.el8.x8 369 kB/s | 62 kB 00:00 2024-03-18T20:16:08.646 INFO:teuthology.orchestra.run.smithi098.stdout:(126/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 3.1 MB/s | 562 kB 00:00 2024-03-18T20:16:08.654 INFO:teuthology.orchestra.run.smithi136.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 853 kB/s | 185 kB 00:00 2024-03-18T20:16:08.713 INFO:teuthology.orchestra.run.smithi136.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2024-03-18T20:16:08.729 INFO:teuthology.orchestra.run.smithi098.stdout:(127/146): openssl-devel-1.1.1k-12.el8_9.x86_64 9.9 MB/s | 2.3 MB 00:00 2024-03-18T20:16:08.780 INFO:teuthology.orchestra.run.smithi098.stdout:(128/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 282 kB/s | 47 kB 00:00 2024-03-18T20:16:08.838 INFO:teuthology.orchestra.run.smithi136.stdout:(46/146): python3-asyncssh-2.7.0-2.el8.noarch.r 949 kB/s | 476 kB 00:00 2024-03-18T20:16:08.922 INFO:teuthology.orchestra.run.smithi098.stdout:(129/146): python3-cryptography-3.2.1-7.el8_9.x 2.0 MB/s | 559 kB 00:00 2024-03-18T20:16:08.930 INFO:teuthology.orchestra.run.smithi136.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 888 kB/s | 81 kB 00:00 2024-03-18T20:16:08.989 INFO:teuthology.orchestra.run.smithi098.stdout:(130/146): python3-packaging-16.8-9.el8.noarch. 186 kB/s | 48 kB 00:00 2024-03-18T20:16:09.005 INFO:teuthology.orchestra.run.smithi136.stdout:(48/146): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-03-18T20:16:09.056 INFO:teuthology.orchestra.run.smithi136.stdout:(49/146): python3-cheroot-8.5.2-1.el8.noarch.rp 430 kB/s | 173 kB 00:00 2024-03-18T20:16:09.081 INFO:teuthology.orchestra.run.smithi136.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2024-03-18T20:16:09.114 INFO:teuthology.orchestra.run.smithi098.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.8 MB/s | 232 kB 00:00 2024-03-18T20:16:09.223 INFO:teuthology.orchestra.run.smithi098.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.7 MB/s | 303 kB 00:00 2024-03-18T20:16:09.348 INFO:teuthology.orchestra.run.smithi098.stdout:(133/146): librbd1-18.2.2-914.g49bdc511.el8.x86 8.4 MB/s | 3.6 MB 00:00 2024-03-18T20:16:09.364 INFO:teuthology.orchestra.run.smithi136.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 172 kB/s | 49 kB 00:00 2024-03-18T20:16:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-03-18T20:16:09.424 INFO:teuthology.orchestra.run.smithi098.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 2.3 MB/s | 179 kB 00:00 2024-03-18T20:16:09.565 INFO:teuthology.orchestra.run.smithi136.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 294 kB/s | 59 kB 00:00 2024-03-18T20:16:09.607 INFO:teuthology.orchestra.run.smithi098.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 424 kB/s | 99 kB 00:00 2024-03-18T20:16:09.716 INFO:teuthology.orchestra.run.smithi098.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 8.5 MB/s | 2.5 MB 00:00 2024-03-18T20:16:09.741 INFO:teuthology.orchestra.run.smithi098.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 768 kB/s | 103 kB 00:00 2024-03-18T20:16:09.748 INFO:teuthology.orchestra.run.smithi136.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 329 kB/s | 60 kB 00:00 2024-03-18T20:16:09.842 INFO:teuthology.orchestra.run.smithi098.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 6.7 MB/s | 854 kB 00:00 2024-03-18T20:16:09.849 INFO:teuthology.orchestra.run.smithi136.stdout:(54/146): python3-kubernetes-11.0.0-6.el8.noarc 1.5 MB/s | 1.2 MB 00:00 2024-03-18T20:16:09.867 INFO:teuthology.orchestra.run.smithi098.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-03-18T20:16:09.899 INFO:teuthology.orchestra.run.smithi136.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 314 kB/s | 16 kB 00:00 2024-03-18T20:16:09.932 INFO:teuthology.orchestra.run.smithi136.stdout:(56/146): python3-cherrypy-18.4.0-1.el8.noarch. 315 kB/s | 384 kB 00:01 2024-03-18T20:16:09.958 INFO:teuthology.orchestra.run.smithi136.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 574 kB/s | 34 kB 00:00 2024-03-18T20:16:10.041 INFO:teuthology.orchestra.run.smithi136.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 770 kB/s | 64 kB 00:00 2024-03-18T20:16:10.042 INFO:teuthology.orchestra.run.smithi098.stdout:(141/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.0 MB/s | 711 kB 00:00 2024-03-18T20:16:10.068 INFO:teuthology.orchestra.run.smithi098.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 254 kB 00:00 2024-03-18T20:16:10.125 INFO:teuthology.orchestra.run.smithi136.stdout:(59/146): python3-simplegeneric-0.8.1-17.el8.no 225 kB/s | 19 kB 00:00 2024-03-18T20:16:10.134 INFO:teuthology.orchestra.run.smithi098.stdout:(143/146): librados2-18.2.2-914.g49bdc511.el8.x 2.7 MB/s | 3.7 MB 00:01 2024-03-18T20:16:10.184 INFO:teuthology.orchestra.run.smithi136.stdout:(60/146): python3-singledispatch-3.4.0.3-18.el8 410 kB/s | 24 kB 00:00 2024-03-18T20:16:10.201 INFO:teuthology.orchestra.run.smithi098.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-03-18T20:16:10.234 INFO:teuthology.orchestra.run.smithi136.stdout:(61/146): python3-tempora-1.14.1-5.el8.noarch.r 586 kB/s | 29 kB 00:00 2024-03-18T20:16:10.235 INFO:teuthology.orchestra.run.smithi098.stdout:(145/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 7.7 MB/s | 1.5 MB 00:00 2024-03-18T20:16:10.284 INFO:teuthology.orchestra.run.smithi136.stdout:(62/146): python3-trustme-0.6.0-4.el8.noarch.rp 552 kB/s | 27 kB 00:00 2024-03-18T20:16:10.344 INFO:teuthology.orchestra.run.smithi098.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-03-18T20:16:10.347 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:16:10.347 INFO:teuthology.orchestra.run.smithi098.stdout:Total 14 MB/s | 212 MB 00:15 2024-03-18T20:16:10.393 INFO:teuthology.orchestra.run.smithi136.stdout:(63/146): python3-routes-2.4.1-12.el8.noarch.rp 427 kB/s | 196 kB 00:00 2024-03-18T20:16:10.452 INFO:teuthology.orchestra.run.smithi136.stdout:(64/146): python3-pecan-1.3.2-9.el8.noarch.rpm 402 kB/s | 283 kB 00:00 2024-03-18T20:16:10.476 INFO:teuthology.orchestra.run.smithi136.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 1.3 MB/s | 255 kB 00:00 2024-03-18T20:16:10.835 INFO:teuthology.orchestra.run.smithi136.stdout:(66/146): python3-webtest-2.0.33-1.el8.noarch.r 238 kB/s | 85 kB 00:00 2024-03-18T20:16:10.861 INFO:teuthology.orchestra.run.smithi136.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 150 kB/s | 61 kB 00:00 2024-03-18T20:16:10.894 INFO:teuthology.orchestra.run.smithi136.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2024-03-18T20:16:10.919 INFO:teuthology.orchestra.run.smithi136.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-03-18T20:16:11.045 INFO:teuthology.orchestra.run.smithi136.stdout:(70/146): python3-webob-1.8.5-1.el8.1.noarch.rp 385 kB/s | 251 kB 00:00 2024-03-18T20:16:11.095 INFO:teuthology.orchestra.run.smithi136.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 102 kB/s | 18 kB 00:00 2024-03-18T20:16:11.170 INFO:teuthology.orchestra.run.smithi136.stdout:(72/146): python3-jwt-1.6.1-2.el8.noarch.rpm 341 kB/s | 43 kB 00:00 2024-03-18T20:16:11.212 INFO:teuthology.orchestra.run.smithi136.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.3 MB/s | 155 kB 00:00 2024-03-18T20:16:11.321 INFO:teuthology.orchestra.run.smithi136.stdout:(74/146): python3-pycparser-2.14-14.el8.noarch. 726 kB/s | 109 kB 00:00 2024-03-18T20:16:11.371 INFO:teuthology.orchestra.run.smithi136.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 269 kB/s | 43 kB 00:00 2024-03-18T20:16:11.405 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-18T20:16:11.454 INFO:teuthology.orchestra.run.smithi136.stdout:(76/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 140 kB/s | 19 kB 00:00 2024-03-18T20:16:11.471 INFO:teuthology.orchestra.run.smithi136.stdout:(77/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 438 kB/s | 44 kB 00:00 2024-03-18T20:16:11.588 INFO:teuthology.orchestra.run.smithi136.stdout:(78/146): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2024-03-18T20:16:11.630 INFO:teuthology.orchestra.run.smithi136.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.4 MB/s | 229 kB 00:00 2024-03-18T20:16:11.706 INFO:teuthology.orchestra.run.smithi136.stdout:(80/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2024-03-18T20:16:11.722 INFO:teuthology.orchestra.run.smithi136.stdout:(81/146): libselinux-devel-2.9-5.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2024-03-18T20:16:11.799 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-18T20:16:11.799 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-18T20:16:11.856 INFO:teuthology.orchestra.run.smithi136.stdout:(82/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 321 kB/s | 48 kB 00:00 2024-03-18T20:16:11.890 INFO:teuthology.orchestra.run.smithi136.stdout:(83/146): libsepol-devel-2.9-3.el8.x86_64.rpm 521 kB/s | 87 kB 00:00 2024-03-18T20:16:11.910 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:16:11.911 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T20:16:11.929 INFO:teuthology.orchestra.run.smithi169.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-18T20:16:11.930 INFO:teuthology.orchestra.run.smithi169.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-18T20:16:11.932 INFO:teuthology.orchestra.run.smithi169.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-18T20:16:11.933 INFO:teuthology.orchestra.run.smithi169.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-18T20:16:11.936 INFO:teuthology.orchestra.run.smithi169.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-18T20:16:11.940 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-18T20:16:11.941 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-18T20:16:11.990 INFO:teuthology.orchestra.run.smithi136.stdout:(84/146): libcom_err-devel-1.45.6-4.el8.x86_64. 386 kB/s | 39 kB 00:00 2024-03-18T20:16:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:(85/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2024-03-18T20:16:12.132 INFO:teuthology.orchestra.run.smithi136.stdout:(86/146): fuse-2.9.7-17.el8.x86_64.rpm 585 kB/s | 83 kB 00:00 2024-03-18T20:16:12.160 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-18T20:16:12.166 INFO:teuthology.orchestra.run.smithi136.stdout:(87/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-03-18T20:16:12.174 INFO:teuthology.orchestra.run.smithi169.stdout:=================================================================================================================================== 2024-03-18T20:16:12.174 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-18T20:16:12.174 INFO:teuthology.orchestra.run.smithi169.stdout:=================================================================================================================================== 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.4 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:18.2.2-914.g49bdc511.el8 ceph 5.4 M 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 854 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-914.g49bdc511.el8 ceph 165 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:18.2.2-914.g49bdc511.el8 ceph 1.6 M 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 138 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 1.7 M 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 7.4 M 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 49 k 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 13 M 2024-03-18T20:16:12.175 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:18.2.2-914.g49bdc511.el8 ceph 47 M 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 263 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 172 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 261 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 18 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 31 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 775 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 139 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-18T20:16:12.176 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:18.2.2-914.g49bdc511.el8 ceph 196 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:18.2.2-914.g49bdc511.el8 ceph 370 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 365 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 107 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 95 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.5 M 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 182 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-18T20:16:12.177 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.7 M 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.6 M 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-18T20:16:12.178 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 22 M 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 23 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:18.2.2-914.g49bdc511.el8 ceph 2.3 M 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 245 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:18.2.2-914.g49bdc511.el8 ceph 4.6 M 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 18 M 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 15 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:18.2.2-914.g49bdc511.el8 ceph 24 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-18T20:16:12.179 INFO:teuthology.orchestra.run.smithi169.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:18.2.2-914.g49bdc511.el8 ceph 142 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 474 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.5 M 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-18T20:16:12.180 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-18T20:16:12.181 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 45 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 126 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-18T20:16:12.182 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-18T20:16:12.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-18T20:16:12.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-18T20:16:12.185 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling module streams: 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql 10 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout:=================================================================================================================================== 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout:Install 130 Packages 2024-03-18T20:16:12.186 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 16 Packages 2024-03-18T20:16:12.187 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:16:12.187 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 212 M 2024-03-18T20:16:12.187 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-18T20:16:12.258 INFO:teuthology.orchestra.run.smithi136.stdout:(88/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-03-18T20:16:12.275 INFO:teuthology.orchestra.run.smithi136.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 920 kB/s | 100 kB 00:00 2024-03-18T20:16:12.334 INFO:teuthology.orchestra.run.smithi136.stdout:(90/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2024-03-18T20:16:12.337 INFO:teuthology.orchestra.run.smithi169.stdout:(1/146): ceph-18.2.2-914.g49bdc511.el8.x86_64.r 56 kB/s | 6.4 kB 00:00 2024-03-18T20:16:12.350 INFO:teuthology.orchestra.run.smithi136.stdout:(91/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 962 kB/s | 73 kB 00:00 2024-03-18T20:16:12.426 INFO:teuthology.orchestra.run.smithi136.stdout:(92/146): NetworkManager-initscripts-updown-1.4 1.5 MB/s | 144 kB 00:00 2024-03-18T20:16:12.518 INFO:teuthology.orchestra.run.smithi136.stdout:(93/146): elfutils-libelf-devel-0.189-3.el8.x86 370 kB/s | 62 kB 00:00 2024-03-18T20:16:12.537 INFO:teuthology.orchestra.run.smithi169.stdout:(2/146): ceph-fuse-18.2.2-914.g49bdc511.el8.x86 4.2 MB/s | 854 kB 00:00 2024-03-18T20:16:12.596 INFO:teuthology.orchestra.run.smithi169.stdout:(3/146): ceph-immutable-object-cache-18.2.2-914 2.8 MB/s | 165 kB 00:00 2024-03-18T20:16:12.618 INFO:teuthology.orchestra.run.smithi136.stdout:(94/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 2.9 MB/s | 562 kB 00:00 2024-03-18T20:16:12.727 INFO:teuthology.orchestra.run.smithi136.stdout:(95/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 11 MB/s | 2.3 MB 00:00 2024-03-18T20:16:12.769 INFO:teuthology.orchestra.run.smithi136.stdout:(96/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 313 kB/s | 47 kB 00:00 2024-03-18T20:16:12.796 INFO:teuthology.orchestra.run.smithi169.stdout:(4/146): ceph-base-18.2.2-914.g49bdc511.el8.x86 9.4 MB/s | 5.4 MB 00:00 2024-03-18T20:16:12.819 INFO:teuthology.orchestra.run.smithi136.stdout:(97/146): python3-cryptography-3.2.1-7.el8_9.x8 6.0 MB/s | 559 kB 00:00 2024-03-18T20:16:12.838 INFO:teuthology.orchestra.run.smithi169.stdout:(5/146): ceph-mds-18.2.2-914.g49bdc511.el8.x86_ 9.5 MB/s | 2.3 MB 00:00 2024-03-18T20:16:12.861 INFO:teuthology.orchestra.run.smithi136.stdout:(98/146): python3-packaging-16.8-9.el8.noarch.r 525 kB/s | 48 kB 00:00 2024-03-18T20:16:12.886 INFO:teuthology.orchestra.run.smithi136.stdout:(99/146): python3-jsonpointer-1.10-11.el8.noarc 299 kB/s | 20 kB 00:00 2024-03-18T20:16:12.939 INFO:teuthology.orchestra.run.smithi169.stdout:(6/146): ceph-mgr-18.2.2-914.g49bdc511.el8.x86_ 11 MB/s | 1.6 MB 00:00 2024-03-18T20:16:12.978 INFO:teuthology.orchestra.run.smithi136.stdout:(100/146): python3-prettytable-0.7.2-14.el8.noa 376 kB/s | 44 kB 00:00 2024-03-18T20:16:12.986 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-18T20:16:12.997 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-18T20:16:13.028 INFO:teuthology.orchestra.run.smithi136.stdout:(101/146): thrift-0.13.0-2.el8.x86_64.rpm 819 kB/s | 1.7 MB 00:02 2024-03-18T20:16:13.053 INFO:teuthology.orchestra.run.smithi136.stdout:(102/146): python3-jsonpatch-1.21-2.el8.noarch. 353 kB/s | 27 kB 00:00 2024-03-18T20:16:13.087 INFO:teuthology.orchestra.run.smithi136.stdout:(103/146): python3-pyasn1-modules-0.3.7-6.el8.n 548 kB/s | 110 kB 00:00 2024-03-18T20:16:13.254 INFO:teuthology.orchestra.run.smithi136.stdout:(104/146): bison-3.0.4-10.el8.x86_64.rpm 4.0 MB/s | 688 kB 00:00 2024-03-18T20:16:13.288 INFO:teuthology.orchestra.run.smithi136.stdout:(105/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 488 kB/s | 126 kB 00:00 2024-03-18T20:16:13.313 INFO:teuthology.orchestra.run.smithi136.stdout:(106/146): python3-werkzeug-0.12.2-4.el8.noarch 1.8 MB/s | 457 kB 00:00 2024-03-18T20:16:13.413 INFO:teuthology.orchestra.run.smithi136.stdout:(107/146): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2024-03-18T20:16:13.447 INFO:teuthology.orchestra.run.smithi136.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 291 kB/s | 39 kB 00:00 2024-03-18T20:16:13.480 INFO:teuthology.orchestra.run.smithi169.stdout:(7/146): ceph-mon-18.2.2-914.g49bdc511.el8.x86_ 7.2 MB/s | 4.6 MB 00:00 2024-03-18T20:16:13.489 INFO:teuthology.orchestra.run.smithi136.stdout:(109/146): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-03-18T20:16:13.547 INFO:teuthology.orchestra.run.smithi136.stdout:(110/146): uuid-1.6.2-43.el8.x86_64.rpm 638 kB/s | 64 kB 00:00 2024-03-18T20:16:13.589 INFO:teuthology.orchestra.run.smithi136.stdout:(111/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-03-18T20:16:13.623 INFO:teuthology.orchestra.run.smithi136.stdout:(112/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 774 kB/s | 103 kB 00:00 2024-03-18T20:16:13.698 INFO:teuthology.orchestra.run.smithi136.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 4.8 MB/s | 538 kB 00:00 2024-03-18T20:16:13.773 INFO:teuthology.orchestra.run.smithi136.stdout:(114/146): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-03-18T20:16:13.824 INFO:teuthology.orchestra.run.smithi136.stdout:(115/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-03-18T20:16:13.899 INFO:teuthology.orchestra.run.smithi136.stdout:(116/146): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2024-03-18T20:16:13.915 INFO:teuthology.orchestra.run.smithi136.stdout:(117/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2024-03-18T20:16:13.966 INFO:teuthology.orchestra.run.smithi136.stdout:(118/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2024-03-18T20:16:14.016 INFO:teuthology.orchestra.run.smithi136.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-03-18T20:16:14.167 INFO:teuthology.orchestra.run.smithi136.stdout:(120/146): openblas-threads-0.3.15-6.el8.x86_64 19 MB/s | 5.2 MB 00:00 2024-03-18T20:16:14.200 INFO:teuthology.orchestra.run.smithi136.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 3.8 MB/s | 714 kB 00:00 2024-03-18T20:16:14.259 INFO:teuthology.orchestra.run.smithi136.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2024-03-18T20:16:14.326 INFO:teuthology.orchestra.run.smithi136.stdout:(123/146): openblas-0.3.15-6.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2024-03-18T20:16:14.339 INFO:teuthology.orchestra.run.smithi169.stdout:(8/146): ceph-osd-18.2.2-914.g49bdc511.el8.x86_ 13 MB/s | 18 MB 00:01 2024-03-18T20:16:14.374 INFO:teuthology.orchestra.run.smithi136.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-03-18T20:16:14.398 INFO:teuthology.orchestra.run.smithi169.stdout:(9/146): ceph-selinux-18.2.2-914.g49bdc511.el8. 418 kB/s | 24 kB 00:00 2024-03-18T20:16:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:(125/146): python3-pytz-2017.2-11.el8.noarch.rp 382 kB/s | 54 kB 00:00 2024-03-18T20:16:14.452 INFO:teuthology.orchestra.run.smithi136.stdout:(126/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-03-18T20:16:14.552 INFO:teuthology.orchestra.run.smithi136.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 10 MB/s | 1.5 MB 00:00 2024-03-18T20:16:14.627 INFO:teuthology.orchestra.run.smithi136.stdout:(128/146): postgresql-contrib-10.23-4.module+el 4.5 MB/s | 812 kB 00:00 2024-03-18T20:16:14.794 INFO:teuthology.orchestra.run.smithi136.stdout:(129/146): python3-scipy-1.0.0-21.module+el8.9. 29 MB/s | 14 MB 00:00 2024-03-18T20:16:14.840 INFO:teuthology.orchestra.run.smithi169.stdout:(10/146): ceph-radosgw-18.2.2-914.g49bdc511.el8 9.6 MB/s | 13 MB 00:01 2024-03-18T20:16:14.886 INFO:teuthology.orchestra.run.smithi136.stdout:(130/146): postgresql-server-10.23-4.module+el8 15 MB/s | 5.1 MB 00:00 2024-03-18T20:16:14.931 INFO:teuthology.orchestra.run.smithi169.stdout:(11/146): cephfs-mirror-18.2.2-914.g49bdc511.el 2.8 MB/s | 261 kB 00:00 2024-03-18T20:16:15.007 INFO:teuthology.orchestra.run.smithi169.stdout:(12/146): libcephfs-devel-18.2.2-914.g49bdc511. 412 kB/s | 31 kB 00:00 2024-03-18T20:16:15.044 INFO:teuthology.orchestra.run.smithi136.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.4 MB/s | 232 kB 00:00 2024-03-18T20:16:15.174 INFO:teuthology.orchestra.run.smithi169.stdout:(13/146): libcephfs2-18.2.2-914.g49bdc511.el8.x 4.5 MB/s | 775 kB 00:00 2024-03-18T20:16:15.186 INFO:teuthology.orchestra.run.smithi136.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.1 MB/s | 303 kB 00:00 2024-03-18T20:16:15.258 INFO:teuthology.orchestra.run.smithi169.stdout:(14/146): libcephsqlite-18.2.2-914.g49bdc511.el 1.7 MB/s | 142 kB 00:00 2024-03-18T20:16:15.333 INFO:teuthology.orchestra.run.smithi169.stdout:(15/146): librados-devel-18.2.2-914.g49bdc511.e 1.8 MB/s | 139 kB 00:00 2024-03-18T20:16:15.353 INFO:teuthology.orchestra.run.smithi136.stdout:(133/146): libmount-2.32.1-43.el8.x86_64.rpm 1.4 MB/s | 237 kB 00:00 2024-03-18T20:16:15.425 INFO:teuthology.orchestra.run.smithi169.stdout:(16/146): libradosstriper1-18.2.2-914.g49bdc511 5.1 MB/s | 474 kB 00:00 2024-03-18T20:16:15.529 INFO:teuthology.orchestra.run.smithi136.stdout:(134/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.0 MB/s | 179 kB 00:00 2024-03-18T20:16:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:(135/146): librados2-18.2.2-914.g49bdc511.el8.x 3.4 MB/s | 3.7 MB 00:01 2024-03-18T20:16:15.746 INFO:teuthology.orchestra.run.smithi136.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 457 kB/s | 99 kB 00:00 2024-03-18T20:16:15.888 INFO:teuthology.orchestra.run.smithi136.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 15 MB/s | 2.5 MB 00:00 2024-03-18T20:16:15.904 INFO:teuthology.orchestra.run.smithi136.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 647 kB/s | 103 kB 00:00 2024-03-18T20:16:16.005 INFO:teuthology.orchestra.run.smithi136.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 7.2 MB/s | 854 kB 00:00 2024-03-18T20:16:16.030 INFO:teuthology.orchestra.run.smithi136.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-03-18T20:16:16.130 INFO:teuthology.orchestra.run.smithi136.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 254 kB 00:00 2024-03-18T20:16:16.158 INFO:teuthology.orchestra.run.smithi169.stdout:(17/146): ceph-common-18.2.2-914.g49bdc511.el8. 5.5 MB/s | 22 MB 00:03 2024-03-18T20:16:16.164 INFO:teuthology.orchestra.run.smithi136.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 5.2 MB/s | 711 kB 00:00 2024-03-18T20:16:16.242 INFO:teuthology.orchestra.run.smithi169.stdout:(18/146): librgw2-18.2.2-914.g49bdc511.el8.x86_ 8.0 MB/s | 6.5 MB 00:00 2024-03-18T20:16:16.267 INFO:teuthology.orchestra.run.smithi169.stdout:(19/146): python3-ceph-argparse-18.2.2-914.g49b 412 kB/s | 45 kB 00:00 2024-03-18T20:16:16.301 INFO:teuthology.orchestra.run.smithi169.stdout:(20/146): python3-ceph-common-18.2.2-914.g49bdc 2.1 MB/s | 126 kB 00:00 2024-03-18T20:16:16.343 INFO:teuthology.orchestra.run.smithi169.stdout:(21/146): python3-cephfs-18.2.2-914.g49bdc511.e 2.5 MB/s | 196 kB 00:00 2024-03-18T20:16:16.368 INFO:teuthology.orchestra.run.smithi169.stdout:(22/146): python3-rados-18.2.2-914.g49bdc511.el 5.4 MB/s | 370 kB 00:00 2024-03-18T20:16:16.381 INFO:teuthology.orchestra.run.smithi136.stdout:(143/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.9 MB/s | 1.5 MB 00:00 2024-03-18T20:16:16.414 INFO:teuthology.orchestra.run.smithi136.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.3 MB/s | 333 kB 00:00 2024-03-18T20:16:16.427 INFO:teuthology.orchestra.run.smithi169.stdout:(23/146): python3-rbd-18.2.2-914.g49bdc511.el8. 4.3 MB/s | 365 kB 00:00 2024-03-18T20:16:16.452 INFO:teuthology.orchestra.run.smithi169.stdout:(24/146): python3-rgw-18.2.2-914.g49bdc511.el8. 1.2 MB/s | 107 kB 00:00 2024-03-18T20:16:16.490 INFO:teuthology.orchestra.run.smithi136.stdout:(145/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-18T20:16:16.502 INFO:teuthology.orchestra.run.smithi169.stdout:(25/146): rbd-fuse-18.2.2-914.g49bdc511.el8.x86 1.2 MB/s | 95 kB 00:00 2024-03-18T20:16:16.594 INFO:teuthology.orchestra.run.smithi169.stdout:(26/146): rbd-nbd-18.2.2-914.g49bdc511.el8.x86_ 1.9 MB/s | 182 kB 00:00 2024-03-18T20:16:16.678 INFO:teuthology.orchestra.run.smithi169.stdout:(27/146): ceph-grafana-dashboards-18.2.2-914.g4 279 kB/s | 23 kB 00:00 2024-03-18T20:16:16.761 INFO:teuthology.orchestra.run.smithi169.stdout:(28/146): ceph-mgr-cephadm-18.2.2-914.g49bdc511 1.6 MB/s | 138 kB 00:00 2024-03-18T20:16:16.842 INFO:teuthology.orchestra.run.smithi169.stdout:(29/146): rbd-mirror-18.2.2-914.g49bdc511.el8.x 8.9 MB/s | 3.5 MB 00:00 2024-03-18T20:16:16.881 INFO:teuthology.orchestra.run.smithi136.stdout:(146/146): librbd1-18.2.2-914.g49bdc511.el8.x86 1.7 MB/s | 3.6 MB 00:02 2024-03-18T20:16:16.886 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:16:16.887 INFO:teuthology.orchestra.run.smithi136.stdout:Total 12 MB/s | 212 MB 00:17 2024-03-18T20:16:17.096 INFO:teuthology.orchestra.run.smithi169.stdout:(30/146): ceph-mgr-dashboard-18.2.2-914.g49bdc5 5.2 MB/s | 1.7 MB 00:00 2024-03-18T20:16:17.187 INFO:teuthology.orchestra.run.smithi169.stdout:(31/146): ceph-mgr-modules-core-18.2.2-914.g49b 2.6 MB/s | 245 kB 00:00 2024-03-18T20:16:17.287 INFO:teuthology.orchestra.run.smithi169.stdout:(32/146): ceph-mgr-rook-18.2.2-914.g49bdc511.el 495 kB/s | 49 kB 00:00 2024-03-18T20:16:17.363 INFO:teuthology.orchestra.run.smithi169.stdout:(33/146): ceph-prometheus-alerts-18.2.2-914.g49 196 kB/s | 15 kB 00:00 2024-03-18T20:16:17.455 INFO:teuthology.orchestra.run.smithi169.stdout:(34/146): ceph-volume-18.2.2-914.g49bdc511.el8. 2.8 MB/s | 263 kB 00:00 2024-03-18T20:16:17.589 INFO:teuthology.orchestra.run.smithi169.stdout:(35/146): ceph-mgr-diskprediction-local-18.2.2- 9.9 MB/s | 7.4 MB 00:00 2024-03-18T20:16:17.614 INFO:teuthology.orchestra.run.smithi169.stdout:(36/146): cephadm-18.2.2-914.g49bdc511.el8.noar 1.1 MB/s | 172 kB 00:00 2024-03-18T20:16:17.647 INFO:teuthology.orchestra.run.smithi169.stdout:(37/146): cephfs-top-18.2.2-914.g49bdc511.el8.n 311 kB/s | 18 kB 00:00 2024-03-18T20:16:17.930 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-18T20:16:18.162 INFO:teuthology.orchestra.run.smithi169.stdout:(38/146): ceph-test-18.2.2-914.g49bdc511.el8.x8 12 MB/s | 47 MB 00:03 2024-03-18T20:16:18.189 INFO:teuthology.orchestra.run.smithi169.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 108 kB/s | 59 kB 00:00 2024-03-18T20:16:18.282 INFO:teuthology.orchestra.run.smithi169.stdout:(40/146): gperftools-libs-2.7-9.el8.x86_64.rpm 459 kB/s | 306 kB 00:00 2024-03-18T20:16:18.324 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-18T20:16:18.324 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-18T20:16:18.340 INFO:teuthology.orchestra.run.smithi169.stdout:(41/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2024-03-18T20:16:18.366 INFO:teuthology.orchestra.run.smithi169.stdout:(42/146): libunwind-1.3.1-3.el8.x86_64.rpm 370 kB/s | 75 kB 00:00 2024-03-18T20:16:18.382 INFO:teuthology.orchestra.run.smithi169.stdout:(43/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 434 kB/s | 44 kB 00:00 2024-03-18T20:16:18.408 INFO:teuthology.orchestra.run.smithi169.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2024-03-18T20:16:18.441 INFO:teuthology.orchestra.run.smithi169.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 279 kB/s | 16 kB 00:00 2024-03-18T20:16:18.458 INFO:teuthology.orchestra.run.smithi169.stdout:(46/146): python3-cachetools-3.1.1-4.el8.noarch 357 kB/s | 33 kB 00:00 2024-03-18T20:16:18.484 INFO:teuthology.orchestra.run.smithi169.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.2 MB/s | 173 kB 00:00 2024-03-18T20:16:18.525 INFO:teuthology.orchestra.run.smithi169.stdout:(48/146): python3-jaraco-6.2-6.el8.noarch.rpm 268 kB/s | 11 kB 00:00 2024-03-18T20:16:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-18T20:16:18.551 INFO:teuthology.orchestra.run.smithi169.stdout:(49/146): python3-google-auth-1.1.1-10.el8.noar 882 kB/s | 81 kB 00:00 2024-03-18T20:16:18.607 INFO:teuthology.orchestra.run.smithi169.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 216 kB/s | 18 kB 00:00 2024-03-18T20:16:18.643 INFO:teuthology.orchestra.run.smithi169.stdout:(51/146): python3-cherrypy-18.4.0-1.el8.noarch. 1.9 MB/s | 384 kB 00:00 2024-03-18T20:16:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-18T20:16:18.668 INFO:teuthology.orchestra.run.smithi169.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 804 kB/s | 49 kB 00:00 2024-03-18T20:16:18.721 INFO:teuthology.orchestra.run.smithi169.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 783 kB/s | 59 kB 00:00 2024-03-18T20:16:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 804 kB/s | 60 kB 00:00 2024-03-18T20:16:18.786 INFO:teuthology.orchestra.run.smithi169.stdout:(55/146): python3-kubernetes-11.0.0-6.el8.noarc 5.0 MB/s | 1.2 MB 00:00 2024-03-18T20:16:18.811 INFO:teuthology.orchestra.run.smithi169.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-03-18T20:16:18.836 INFO:teuthology.orchestra.run.smithi169.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-03-18T20:16:18.862 INFO:teuthology.orchestra.run.smithi169.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 445 kB/s | 34 kB 00:00 2024-03-18T20:16:18.887 INFO:teuthology.orchestra.run.smithi169.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 2.5 MB/s | 196 kB 00:00 2024-03-18T20:16:18.902 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:18.912 INFO:teuthology.orchestra.run.smithi169.stdout:(60/146): python3-rsa-4.9-2.el8.noarch.rpm 851 kB/s | 64 kB 00:00 2024-03-18T20:16:18.937 INFO:teuthology.orchestra.run.smithi169.stdout:(61/146): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2024-03-18T20:16:18.963 INFO:teuthology.orchestra.run.smithi169.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 317 kB/s | 24 kB 00:00 2024-03-18T20:16:18.988 INFO:teuthology.orchestra.run.smithi169.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2024-03-18T20:16:19.013 INFO:teuthology.orchestra.run.smithi169.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 364 kB/s | 27 kB 00:00 2024-03-18T20:16:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:19.130 INFO:teuthology.orchestra.run.smithi169.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 1.5 MB/s | 255 kB 00:00 2024-03-18T20:16:19.155 INFO:teuthology.orchestra.run.smithi169.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 432 kB/s | 61 kB 00:00 2024-03-18T20:16:19.172 INFO:teuthology.orchestra.run.smithi169.stdout:(67/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2024-03-18T20:16:19.197 INFO:teuthology.orchestra.run.smithi169.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2024-03-18T20:16:19.223 INFO:teuthology.orchestra.run.smithi169.stdout:(69/146): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2024-03-18T20:16:19.248 INFO:teuthology.orchestra.run.smithi169.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-03-18T20:16:19.307 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-18T20:16:19.374 INFO:teuthology.orchestra.run.smithi169.stdout:(71/146): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2024-03-18T20:16:19.432 INFO:teuthology.orchestra.run.smithi169.stdout:(72/146): python3-prettytable-0.7.2-14.el8.noar 247 kB/s | 44 kB 00:00 2024-03-18T20:16:19.502 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:19.507 INFO:teuthology.orchestra.run.smithi169.stdout:(73/146): python3-pyasn1-modules-0.3.7-6.el8.no 825 kB/s | 110 kB 00:00 2024-03-18T20:16:19.517 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-18T20:16:19.528 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-18T20:16:19.541 INFO:teuthology.orchestra.run.smithi169.stdout:(74/146): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2024-03-18T20:16:19.566 INFO:teuthology.orchestra.run.smithi169.stdout:(75/146): python3-jsonpatch-1.21-2.el8.noarch.r 198 kB/s | 27 kB 00:00 2024-03-18T20:16:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:19.600 INFO:teuthology.orchestra.run.smithi169.stdout:(76/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2024-03-18T20:16:19.750 INFO:teuthology.orchestra.run.smithi169.stdout:(77/146): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2024-03-18T20:16:19.776 INFO:teuthology.orchestra.run.smithi169.stdout:(78/146): bison-3.0.4-10.el8.x86_64.rpm 3.3 MB/s | 688 kB 00:00 2024-03-18T20:16:19.817 INFO:teuthology.orchestra.run.smithi169.stdout:(79/146): python3-werkzeug-0.12.2-4.el8.noarch. 1.6 MB/s | 457 kB 00:00 2024-03-18T20:16:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:19.951 INFO:teuthology.orchestra.run.smithi169.stdout:(80/146): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-03-18T20:16:19.976 INFO:teuthology.orchestra.run.smithi169.stdout:(81/146): python3-markupsafe-0.23-19.el8.x86_64 194 kB/s | 39 kB 00:00 2024-03-18T20:16:20.001 INFO:teuthology.orchestra.run.smithi169.stdout:(82/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-03-18T20:16:20.052 INFO:teuthology.orchestra.run.smithi169.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 638 kB/s | 64 kB 00:00 2024-03-18T20:16:20.169 INFO:teuthology.orchestra.run.smithi169.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 539 kB/s | 103 kB 00:00 2024-03-18T20:16:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:20.211 INFO:teuthology.orchestra.run.smithi169.stdout:(85/146): python3-jinja2-2.10.1-3.el8.noarch.rp 3.3 MB/s | 538 kB 00:00 2024-03-18T20:16:20.346 INFO:teuthology.orchestra.run.smithi169.stdout:(86/146): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-03-18T20:16:20.386 INFO:teuthology.orchestra.run.smithi169.stdout:(87/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-03-18T20:16:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-18T20:16:20.478 INFO:teuthology.orchestra.run.smithi169.stdout:(88/146): python3-numpy-1.14.3-10.el8.x86_64.rp 12 MB/s | 3.7 MB 00:00 2024-03-18T20:16:20.504 INFO:teuthology.orchestra.run.smithi169.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.4 MB/s | 225 kB 00:00 2024-03-18T20:16:20.554 INFO:teuthology.orchestra.run.smithi169.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.1 MB/s | 354 kB 00:00 2024-03-18T20:16:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-18T20:16:20.596 INFO:teuthology.orchestra.run.smithi169.stdout:(91/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2024-03-18T20:16:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-18T20:16:20.721 INFO:teuthology.orchestra.run.smithi169.stdout:(92/146): openblas-threads-0.3.15-6.el8.x86_64. 21 MB/s | 5.2 MB 00:00 2024-03-18T20:16:20.754 INFO:teuthology.orchestra.run.smithi169.stdout:(93/146): automake-1.16.1-8.el8.noarch.rpm 4.4 MB/s | 714 kB 00:00 2024-03-18T20:16:20.821 INFO:teuthology.orchestra.run.smithi169.stdout:(94/146): jq-1.6-7.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2024-03-18T20:16:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-03-18T20:16:20.913 INFO:teuthology.orchestra.run.smithi169.stdout:(95/146): openblas-0.3.15-6.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2024-03-18T20:16:20.939 INFO:teuthology.orchestra.run.smithi169.stdout:(96/146): libpq-13.11-1.el8.x86_64.rpm 911 kB/s | 198 kB 00:00 2024-03-18T20:16:20.997 INFO:teuthology.orchestra.run.smithi169.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 309 kB/s | 54 kB 00:00 2024-03-18T20:16:21.039 INFO:teuthology.orchestra.run.smithi169.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-03-18T20:16:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:21.123 INFO:teuthology.orchestra.run.smithi169.stdout:(99/146): postgresql-10.23-4.module+el8.9.0+213 12 MB/s | 1.5 MB 00:00 2024-03-18T20:16:21.181 INFO:teuthology.orchestra.run.smithi169.stdout:(100/146): postgresql-contrib-10.23-4.module+el 5.6 MB/s | 812 kB 00:00 2024-03-18T20:16:21.340 INFO:teuthology.orchestra.run.smithi169.stdout:(101/146): postgresql-server-10.23-4.module+el8 23 MB/s | 5.1 MB 00:00 2024-03-18T20:16:21.365 INFO:teuthology.orchestra.run.smithi169.stdout:(102/146): python3-packaging-16.8-9.el8.noarch. 262 kB/s | 48 kB 00:00 2024-03-18T20:16:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:21.524 INFO:teuthology.orchestra.run.smithi169.stdout:(103/146): python3-scipy-1.0.0-21.module+el8.9. 22 MB/s | 14 MB 00:00 2024-03-18T20:16:21.549 INFO:teuthology.orchestra.run.smithi169.stdout:(104/146): libverto-devel-0.3.0-5.el8.x86_64.rp 86 kB/s | 18 kB 00:00 2024-03-18T20:16:21.575 INFO:teuthology.orchestra.run.smithi169.stdout:(105/146): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-03-18T20:16:21.600 INFO:teuthology.orchestra.run.smithi169.stdout:(106/146): python3-oauthlib-2.1.0-1.el8.noarch. 2.0 MB/s | 155 kB 00:00 2024-03-18T20:16:21.625 INFO:teuthology.orchestra.run.smithi169.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 1.4 MB/s | 109 kB 00:00 2024-03-18T20:16:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:21.692 INFO:teuthology.orchestra.run.smithi169.stdout:(108/146): python3-requests-oauthlib-1.0.0-1.el 365 kB/s | 43 kB 00:00 2024-03-18T20:16:21.726 INFO:teuthology.orchestra.run.smithi169.stdout:(109/146): libaio-devel-0.3.112-1.el8.x86_64.rp 149 kB/s | 19 kB 00:00 2024-03-18T20:16:21.792 INFO:teuthology.orchestra.run.smithi169.stdout:(110/146): pcre2-devel-10.32-2.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2024-03-18T20:16:21.809 INFO:teuthology.orchestra.run.smithi169.stdout:(111/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 238 kB/s | 44 kB 00:00 2024-03-18T20:16:21.835 INFO:teuthology.orchestra.run.smithi169.stdout:(112/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2024-03-18T20:16:21.893 INFO:teuthology.orchestra.run.smithi169.stdout:(113/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-03-18T20:16:21.927 INFO:teuthology.orchestra.run.smithi169.stdout:(114/146): keyutils-libs-devel-1.5.10-9.el8.x86 523 kB/s | 48 kB 00:00 2024-03-18T20:16:21.977 INFO:teuthology.orchestra.run.smithi169.stdout:(115/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.2 MB/s | 200 kB 00:00 2024-03-18T20:16:22.002 INFO:teuthology.orchestra.run.smithi169.stdout:(116/146): libsepol-devel-2.9-3.el8.x86_64.rpm 799 kB/s | 87 kB 00:00 2024-03-18T20:16:22.053 INFO:teuthology.orchestra.run.smithi169.stdout:(117/146): libcom_err-devel-1.45.6-4.el8.x86_64 514 kB/s | 39 kB 00:00 2024-03-18T20:16:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:22.099 INFO:teuthology.orchestra.run.smithi169.stdout:(118/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 335 kB/s | 58 kB 00:00 2024-03-18T20:16:22.128 INFO:teuthology.orchestra.run.smithi169.stdout:(119/146): fuse-2.9.7-17.el8.x86_64.rpm 661 kB/s | 83 kB 00:00 2024-03-18T20:16:22.204 INFO:teuthology.orchestra.run.smithi169.stdout:(120/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-03-18T20:16:22.229 INFO:teuthology.orchestra.run.smithi169.stdout:(121/146): libuuid-devel-2.32.1-43.el8.x86_64.r 996 kB/s | 100 kB 00:00 2024-03-18T20:16:22.246 INFO:teuthology.orchestra.run.smithi169.stdout:(122/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-03-18T20:16:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:22.321 INFO:teuthology.orchestra.run.smithi169.stdout:(123/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 790 kB/s | 73 kB 00:00 2024-03-18T20:16:22.338 INFO:teuthology.orchestra.run.smithi169.stdout:(124/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2024-03-18T20:16:22.391 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:22.446 INFO:teuthology.orchestra.run.smithi169.stdout:(125/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 5.1 MB/s | 562 kB 00:00 2024-03-18T20:16:22.488 INFO:teuthology.orchestra.run.smithi169.stdout:(126/146): NetworkManager-initscripts-updown-1. 594 kB/s | 144 kB 00:00 2024-03-18T20:16:22.564 INFO:teuthology.orchestra.run.smithi169.stdout:(127/146): elfutils-libelf-devel-0.189-3.el8.x8 254 kB/s | 62 kB 00:00 2024-03-18T20:16:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:22.606 INFO:teuthology.orchestra.run.smithi169.stdout:(128/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 403 kB/s | 47 kB 00:00 2024-03-18T20:16:22.656 INFO:teuthology.orchestra.run.smithi169.stdout:(129/146): openssl-devel-1.1.1k-12.el8_9.x86_64 11 MB/s | 2.3 MB 00:00 2024-03-18T20:16:22.661 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:22.690 INFO:teuthology.orchestra.run.smithi169.stdout:(130/146): python3-cryptography-3.2.1-7.el8_9.x 4.4 MB/s | 559 kB 00:00 2024-03-18T20:16:22.841 INFO:teuthology.orchestra.run.smithi169.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.5 MB/s | 232 kB 00:00 2024-03-18T20:16:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:16:23.057 INFO:teuthology.orchestra.run.smithi169.stdout:(132/146): librados2-18.2.2-914.g49bdc511.el8.x 8.1 MB/s | 3.7 MB 00:00 2024-03-18T20:16:23.116 INFO:teuthology.orchestra.run.smithi169.stdout:(133/146): librbd1-18.2.2-914.g49bdc511.el8.x86 7.8 MB/s | 3.6 MB 00:00 2024-03-18T20:16:23.141 INFO:teuthology.orchestra.run.smithi169.stdout:(134/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.0 MB/s | 303 kB 00:00 2024-03-18T20:16:23.183 INFO:teuthology.orchestra.run.smithi169.stdout:(135/146): libmount-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 237 kB 00:00 2024-03-18T20:16:23.216 INFO:teuthology.orchestra.run.smithi169.stdout:(136/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.8 MB/s | 179 kB 00:00 2024-03-18T20:16:23.226 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:23.258 INFO:teuthology.orchestra.run.smithi169.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 845 kB/s | 99 kB 00:00 2024-03-18T20:16:23.310 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:23.350 INFO:teuthology.orchestra.run.smithi169.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 15 MB/s | 2.5 MB 00:00 2024-03-18T20:16:23.367 INFO:teuthology.orchestra.run.smithi169.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 682 kB/s | 103 kB 00:00 2024-03-18T20:16:23.434 INFO:teuthology.orchestra.run.smithi169.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 4.8 MB/s | 854 kB 00:00 2024-03-18T20:16:23.484 INFO:teuthology.orchestra.run.smithi169.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 254 kB 00:00 2024-03-18T20:16:23.509 INFO:teuthology.orchestra.run.smithi169.stdout:(142/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-03-18T20:16:23.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-18T20:16:23.576 INFO:teuthology.orchestra.run.smithi169.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.9 MB/s | 711 kB 00:00 2024-03-18T20:16:23.652 INFO:teuthology.orchestra.run.smithi169.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-03-18T20:16:23.677 INFO:teuthology.orchestra.run.smithi169.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.9 MB/s | 333 kB 00:00 2024-03-18T20:16:23.735 INFO:teuthology.orchestra.run.smithi169.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 6.7 MB/s | 1.1 MB 00:00 2024-03-18T20:16:23.738 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:16:23.739 INFO:teuthology.orchestra.run.smithi169.stdout:Total 18 MB/s | 212 MB 00:11 2024-03-18T20:16:23.796 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-18T20:16:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-18T20:16:24.305 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-18T20:16:24.580 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-18T20:16:24.743 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:24.802 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-18T20:16:24.846 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:25.196 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-18T20:16:25.196 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-18T20:16:25.276 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-18T20:16:25.418 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-18T20:16:25.423 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-18T20:16:25.554 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-18T20:16:25.585 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-18T20:16:25.817 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:26.211 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-18T20:16:26.243 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-18T20:16:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-18T20:16:26.386 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-18T20:16:26.395 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-18T20:16:26.411 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:26.984 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-18T20:16:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-03-18T20:16:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-18T20:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-18T20:16:27.844 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-18T20:16:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-03-18T20:16:27.943 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-18T20:16:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-03-18T20:16:28.177 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-03-18T20:16:28.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-03-18T20:16:28.586 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-03-18T20:16:28.802 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:29.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-03-18T20:16:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-18T20:16:29.400 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 31/162 2024-03-18T20:16:29.526 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 32/162 2024-03-18T20:16:29.727 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:29.864 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:29.919 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 33/162 2024-03-18T20:16:30.212 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:16:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:30.612 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:30.850 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-18T20:16:30.908 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-03-18T20:16:31.123 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-18T20:16:31.230 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-03-18T20:16:31.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-18T20:16:31.496 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-03-18T20:16:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-18T20:16:31.723 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-18T20:16:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-18T20:16:32.063 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-18T20:16:32.198 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-18T20:16:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:32.331 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:32.465 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:32.526 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:32.684 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-18T20:16:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-18T20:16:32.846 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-18T20:16:33.045 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:33.067 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-18T20:16:33.146 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-18T20:16:33.320 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:33.357 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-18T20:16:33.390 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-18T20:16:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-18T20:16:33.510 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-18T20:16:33.634 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-18T20:16:33.645 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-18T20:16:33.776 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-03-18T20:16:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-18T20:16:33.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-18T20:16:33.882 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-18T20:16:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-18T20:16:34.087 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-03-18T20:16:34.210 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-18T20:16:34.261 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:34.404 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-18T20:16:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 41/162 2024-03-18T20:16:34.609 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-18T20:16:34.645 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-18T20:16:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 42/162 2024-03-18T20:16:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:34.861 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:35.218 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-18T20:16:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:35.328 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:16:35.409 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-18T20:16:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-18T20:16:35.548 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-18T20:16:35.673 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:16:35.869 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:35.892 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:35.969 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:16:36.093 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-18T20:16:36.246 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:16:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:36.373 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-18T20:16:36.536 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-18T20:16:36.609 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-18T20:16:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:36.817 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-18T20:16:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-18T20:16:36.894 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-18T20:16:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.7-17.el8.x86_64 50/162 2024-03-18T20:16:37.056 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-18T20:16:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 31/162 2024-03-18T20:16:37.222 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-18T20:16:37.237 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 51/162 2024-03-18T20:16:37.252 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 32/162 2024-03-18T20:16:37.420 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 52/162 2024-03-18T20:16:37.555 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 33/162 2024-03-18T20:16:37.598 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-18T20:16:37.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 53/162 2024-03-18T20:16:37.792 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-18T20:16:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 54/162 2024-03-18T20:16:37.828 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-18T20:16:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 55/162 2024-03-18T20:16:38.136 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:16:38.277 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:38.341 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/162 2024-03-18T20:16:38.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/162 2024-03-18T20:16:38.510 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-18T20:16:38.622 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/162 2024-03-18T20:16:38.635 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-18T20:16:38.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-18T20:16:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 58/162 2024-03-18T20:16:39.043 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 59/162 2024-03-18T20:16:39.423 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 60/162 2024-03-18T20:16:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-18T20:16:39.536 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-18T20:16:39.611 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-18T20:16:39.702 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 61/162 2024-03-18T20:16:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-18T20:16:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/162 2024-03-18T20:16:40.191 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 63/162 2024-03-18T20:16:40.248 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-18T20:16:40.384 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-18T20:16:40.493 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-03-18T20:16:40.651 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/162 2024-03-18T20:16:40.884 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-18T20:16:41.187 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-03-18T20:16:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-18T20:16:41.363 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/162 2024-03-18T20:16:41.475 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-03-18T20:16:41.517 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-18T20:16:41.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/162 2024-03-18T20:16:41.634 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-03-18T20:16:41.794 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-18T20:16:41.805 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-03-18T20:16:41.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 68/162 2024-03-18T20:16:42.008 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 69/162 2024-03-18T20:16:42.068 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-18T20:16:42.111 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-03-18T20:16:42.236 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 70/162 2024-03-18T20:16:42.312 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 41/162 2024-03-18T20:16:42.390 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-03-18T20:16:42.416 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-18T20:16:42.497 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 42/162 2024-03-18T20:16:42.507 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-18T20:16:42.645 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-18T20:16:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:42.817 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-03-18T20:16:42.833 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 31/162 2024-03-18T20:16:42.962 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 32/162 2024-03-18T20:16:43.041 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-03-18T20:16:43.112 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:43.179 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 33/162 2024-03-18T20:16:43.291 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-03-18T20:16:43.347 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-18T20:16:43.359 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-03-18T20:16:43.540 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-03-18T20:16:43.585 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-18T20:16:43.724 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-03-18T20:16:43.816 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:43.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-03-18T20:16:44.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-18T20:16:44.136 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-03-18T20:16:44.263 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-18T20:16:44.272 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:44.473 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-03-18T20:16:44.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-18T20:16:44.641 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:44.765 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-03-18T20:16:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-18T20:16:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-18T20:16:44.906 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-18T20:16:44.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-18T20:16:45.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-18T20:16:45.177 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-18T20:16:45.191 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-03-18T20:16:45.407 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-18T20:16:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-03-18T20:16:45.603 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-18T20:16:45.615 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-03-18T20:16:45.683 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-03-18T20:16:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-18T20:16:45.897 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-03-18T20:16:46.052 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-18T20:16:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-03-18T20:16:46.221 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-18T20:16:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-03-18T20:16:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-18T20:16:46.517 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-16.8-9.el8.noarch 56/162 2024-03-18T20:16:46.627 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-18T20:16:46.678 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-18T20:16:46.753 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-18T20:16:46.838 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-18T20:16:46.985 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-03-18T20:16:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-18T20:16:47.085 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-18T20:16:47.187 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-18T20:16:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-18T20:16:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-18T20:16:47.427 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-18T20:16:47.461 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-18T20:16:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-18T20:16:47.661 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-18T20:16:47.699 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 41/162 2024-03-18T20:16:47.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-18T20:16:47.874 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 42/162 2024-03-18T20:16:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-18T20:16:48.117 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:48.196 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-18T20:16:48.344 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-18T20:16:48.393 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:48.447 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-18T20:16:48.586 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-18T20:16:48.688 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-18T20:16:48.753 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-18T20:16:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-18T20:16:48.878 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-18T20:16:48.976 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-18T20:16:49.092 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-18T20:16:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-18T20:16:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-18T20:16:49.553 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-18T20:16:49.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-18T20:16:49.689 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-18T20:16:49.859 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:49.877 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-18T20:16:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-18T20:16:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-18T20:16:50.063 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-03-18T20:16:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-18T20:16:50.238 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-03-18T20:16:50.255 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-18T20:16:50.445 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-03-18T20:16:50.448 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-18T20:16:50.611 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-03-18T20:16:50.761 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-18T20:16:50.837 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-03-18T20:16:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-18T20:16:51.035 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-18T20:16:51.131 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:16:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-03-18T20:16:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:16:51.265 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-18T20:16:51.390 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 104/162 2024-03-18T20:16:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-03-18T20:16:51.508 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-18T20:16:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-03-18T20:16:51.752 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-18T20:16:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-03-18T20:16:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-03-18T20:16:52.032 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-18T20:16:52.230 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-18T20:16:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-03-18T20:16:52.417 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-18T20:16:52.686 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 60/162 2024-03-18T20:16:52.739 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-18T20:16:52.983 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-03-18T20:16:53.012 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-18T20:16:53.183 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 62/162 2024-03-18T20:16:53.243 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-18T20:16:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/162 2024-03-18T20:16:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-18T20:16:53.633 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/162 2024-03-18T20:16:53.661 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-18T20:16:53.863 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/162 2024-03-18T20:16:53.917 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-18T20:16:54.092 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/162 2024-03-18T20:16:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-18T20:16:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-03-18T20:16:54.434 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-18T20:16:54.524 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-03-18T20:16:54.603 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-18T20:16:54.802 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-03-18T20:16:54.893 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-18T20:16:54.987 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-16.8-9.el8.noarch 70/162 2024-03-18T20:16:55.034 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:16:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-18T20:16:55.200 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-18T20:16:55.274 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-18T20:16:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:16:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-18T20:16:55.574 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-03-18T20:16:55.649 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-18T20:16:55.834 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-03-18T20:16:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:16:55.946 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-18T20:16:56.058 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-03-18T20:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:16:56.151 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-03-18T20:16:56.154 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-18T20:16:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:16:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-03-18T20:16:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:16:56.387 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-18T20:16:56.507 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-03-18T20:16:56.718 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-03-18T20:16:56.900 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-18T20:16:57.104 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-18T20:16:57.405 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-18T20:16:57.480 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-18T20:16:57.631 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-18T20:16:57.796 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-18T20:16:57.994 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-18T20:16:58.183 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-18T20:16:58.412 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-18T20:16:58.440 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-18T20:16:58.678 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-18T20:16:58.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-18T20:16:58.885 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-18T20:16:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-18T20:16:59.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-18T20:16:59.303 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-18T20:16:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-18T20:16:59.511 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-18T20:16:59.604 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-18T20:16:59.705 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-18T20:16:59.968 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-18T20:17:00.240 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-18T20:17:00.414 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-18T20:17:00.420 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-18T20:17:00.637 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:17:00.673 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-18T20:17:00.695 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:17:00.921 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 104/162 2024-03-18T20:17:01.744 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-18T20:17:02.070 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-18T20:17:02.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-18T20:17:02.609 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-18T20:17:02.865 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-18T20:17:03.073 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-18T20:17:03.891 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-18T20:17:04.065 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:17:04.164 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 103/162 2024-03-18T20:17:04.348 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 104/162 2024-03-18T20:17:04.704 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:04.987 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:05.858 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:17:05.930 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:17:06.128 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:17:08.033 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:08.265 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:08.845 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/162 2024-03-18T20:17:09.126 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:17:09.210 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/162 2024-03-18T20:17:09.380 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /run 2024-03-18T20:17:15.916 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-18T20:17:15.917 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-18T20:17:15.917 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-18T20:17:15.917 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-18T20:17:15.917 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2024-03-18T20:17:15.917 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:17:16.345 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:17.910 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:18.331 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:19.162 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 113/162 2024-03-18T20:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 114/162 2024-03-18T20:17:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:23.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:23.403 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:23.763 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 118/162 2024-03-18T20:17:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:24.024 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:24.177 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:24.188 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 121/162 2024-03-18T20:17:25.587 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:17:25.587 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-03-18T20:17:25.587 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /run 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2024-03-18T20:17:25.588 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:17:25.635 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:25.669 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:25.869 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:25.913 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:26.457 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:27.562 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:27.999 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:28.157 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:28.613 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:28.781 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:28.800 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:28.918 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/162 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /run 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-03-18T20:17:28.919 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:17:29.303 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:30.757 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:30.871 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/162 2024-03-18T20:17:30.964 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:31.072 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:31.084 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:31.095 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 113/162 2024-03-18T20:17:31.293 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/162 2024-03-18T20:17:31.315 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 114/162 2024-03-18T20:17:31.461 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:31.558 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 126/162 2024-03-18T20:17:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:31.785 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:31.895 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-03-18T20:17:31.918 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/162 2024-03-18T20:17:32.086 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:32.104 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/162 2024-03-18T20:17:32.127 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:32.146 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-03-18T20:17:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-03-18T20:17:33.091 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-03-18T20:17:33.313 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-03-18T20:17:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 131/162 2024-03-18T20:17:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 118/162 2024-03-18T20:17:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-18T20:17:34.036 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:34.050 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:34.075 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:34.097 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-18T20:17:34.199 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 112/162 2024-03-18T20:17:34.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 134/162 2024-03-18T20:17:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:34.324 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 113/162 2024-03-18T20:17:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 114/162 2024-03-18T20:17:34.719 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:34.941 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 115/162 2024-03-18T20:17:35.280 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:35.564 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 121/162 2024-03-18T20:17:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:36.080 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:36.124 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:36.248 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 116/162 2024-03-18T20:17:36.338 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:36.410 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:36.502 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 117/162 2024-03-18T20:17:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:36.810 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:36.875 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 118/162 2024-03-18T20:17:36.913 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:37.088 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:37.127 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 119/162 2024-03-18T20:17:37.264 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-18T20:17:37.268 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:37.280 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 120/162 2024-03-18T20:17:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-18T20:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-18T20:17:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 139/162 2024-03-18T20:17:38.351 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 140/162 2024-03-18T20:17:38.537 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 121/162 2024-03-18T20:17:38.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 141/162 2024-03-18T20:17:38.713 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:38.748 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 122/162 2024-03-18T20:17:38.927 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-18T20:17:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:39.003 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 123/162 2024-03-18T20:17:39.092 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-18T20:17:39.103 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-18T20:17:39.160 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-03-18T20:17:39.507 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 124/162 2024-03-18T20:17:39.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-03-18T20:17:39.615 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-03-18T20:17:39.726 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-03-18T20:17:39.839 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-03-18T20:17:40.004 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-18T20:17:40.185 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-18T20:17:40.379 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-18T20:17:40.379 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:40.653 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:40.653 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:40.888 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:40.888 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:41.173 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:41.348 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:41.393 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-18T20:17:41.577 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:41.843 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-18T20:17:41.844 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:42.086 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:42.086 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:42.328 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:42.328 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:42.370 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 126/162 2024-03-18T20:17:42.586 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:42.586 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:42.691 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-18T20:17:42.835 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:42.836 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:42.993 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-18T20:17:43.103 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-18T20:17:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:43.431 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 129/162 2024-03-18T20:17:43.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-18T20:17:43.978 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-18T20:17:43.992 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 1/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 3/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 4/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 5/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 6/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 7/162 2024-03-18T20:17:44.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 8/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 13/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 14/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 15/162 2024-03-18T20:17:44.158 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 16/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 17/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 18/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 20/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 21/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 22/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 23/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 24/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 25/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 26/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 27/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 28/162 2024-03-18T20:17:44.159 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 29/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 30/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 31/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 32/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 33/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 34/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 35/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 36/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 37/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 38/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-18T20:17:44.160 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-18T20:17:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-18T20:17:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-03-18T20:17:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-03-18T20:17:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-03-18T20:17:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/162 2024-03-18T20:17:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 123/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 124/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 126/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 128/162 2024-03-18T20:17:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 129/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 131/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 133/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-18T20:17:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-18T20:17:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-18T20:17:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-18T20:17:44.171 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-18T20:17:44.178 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 125/162 2024-03-18T20:17:44.222 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-18T20:17:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-18T20:17:44.661 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 126/162 2024-03-18T20:17:44.662 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 134/162 2024-03-18T20:17:44.961 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-03-18T20:17:45.176 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-03-18T20:17:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-03-18T20:17:46.145 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-03-18T20:17:46.440 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-03-18T20:17:46.655 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 131/162 2024-03-18T20:17:46.902 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-18T20:17:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:47.079 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-18T20:17:47.267 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 134/162 2024-03-18T20:17:47.278 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:47.326 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:47.786 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-18T20:17:48.223 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-18T20:17:48.489 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-18T20:17:48.735 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 139/162 2024-03-18T20:17:48.943 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 140/162 2024-03-18T20:17:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 141/162 2024-03-18T20:17:49.537 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:49.587 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-18T20:17:49.859 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-18T20:17:49.866 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-18T20:17:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-18T20:17:50.193 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-18T20:17:50.207 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-18T20:17:50.323 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-18T20:17:50.376 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-18T20:17:50.385 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-18T20:17:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-18T20:17:50.717 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-18T20:17:50.785 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-18T20:17:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-18T20:17:51.003 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-18T20:17:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 139/162 2024-03-18T20:17:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-18T20:17:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:51.391 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 140/162 2024-03-18T20:17:51.471 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:51.471 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 141/162 2024-03-18T20:17:51.747 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:51.747 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-18T20:17:52.041 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:52.157 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-18T20:17:52.168 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-18T20:17:52.286 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-18T20:17:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-03-18T20:17:52.512 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:52.607 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-03-18T20:17:52.687 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-03-18T20:17:52.769 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-18T20:17:52.769 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-03-18T20:17:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-03-18T20:17:53.062 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:53.062 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:53.106 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-18T20:17:53.309 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-18T20:17:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:53.486 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-18T20:17:53.486 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:53.653 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:53.653 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:53.752 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-18T20:17:53.752 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:53.919 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:53.920 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:53.994 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-18T20:17:53.994 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:54.280 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-18T20:17:54.370 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-18T20:17:54.370 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:54.492 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-18T20:17:54.650 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:54.908 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-18T20:17:54.908 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:55.168 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-18T20:17:55.168 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 1/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 3/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 4/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 5/162 2024-03-18T20:17:55.316 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 6/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 7/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 8/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 13/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 14/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 15/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 16/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 17/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 18/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:17:55.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 20/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 21/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 22/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 23/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 24/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 25/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 26/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 27/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 28/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 29/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 30/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 31/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 32/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 33/162 2024-03-18T20:17:55.318 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 34/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 35/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 36/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 37/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 38/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-18T20:17:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-18T20:17:55.320 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-18T20:17:55.321 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-18T20:17:55.322 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-18T20:17:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 99/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-03-18T20:17:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-03-18T20:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 131/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 133/162 2024-03-18T20:17:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-18T20:17:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-18T20:17:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-18T20:17:55.401 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-18T20:17:55.402 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:55.685 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-18T20:17:55.685 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:55.909 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-18T20:17:55.909 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:56.159 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-18T20:17:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-18T20:17:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:57.115 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-18T20:17:57.115 INFO:teuthology.orchestra.run.smithi098.stdout:Installed products updated. 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 1/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 3/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 4/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 5/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 6/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 7/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 8/162 2024-03-18T20:17:57.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 9/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 10/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 11/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 12/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 13/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 14/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 15/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 16/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 17/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 18/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 19/162 2024-03-18T20:17:57.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 20/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 21/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 22/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 23/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 24/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 25/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 26/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 27/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 28/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 29/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 30/162 2024-03-18T20:17:57.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 31/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 32/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 33/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 34/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 35/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 36/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 37/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 38/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-18T20:17:57.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-18T20:17:57.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-18T20:17:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-03-18T20:17:57.320 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-03-18T20:17:57.321 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-03-18T20:17:57.322 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 103/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-03-18T20:17:57.323 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/162 2024-03-18T20:17:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 125/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 126/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 129/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 130/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 131/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 133/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-18T20:17:57.325 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-18T20:17:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-18T20:17:57.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-18T20:17:57.328 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-18T20:17:57.750 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-18T20:17:57.751 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.1-8.el8.noarch 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.0.4-10.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.752 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-18T20:17:57.753 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.1-9.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-7.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-18T20:17:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: libpq-13.11-1.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-18T20:17:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-18T20:17:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-18T20:17:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-18T20:17:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-2.6-1.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-18T20:17:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-18T20:17:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-18T20:17:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:17:57.762 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-18T20:18:00.362 DEBUG:teuthology.parallel:result is None 2024-03-18T20:18:10.244 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-18T20:18:10.244 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: automake-1.16.1-8.el8.noarch 2024-03-18T20:18:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: bison-3.0.4-10.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: flex-2.6.1-9.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-7.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libpq-13.11-1.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-18T20:18:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-18T20:18:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-18T20:18:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-2.6-1.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-18T20:18:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-18T20:18:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-18T20:18:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-18T20:18:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-18T20:18:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-18T20:18:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-18T20:18:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-18T20:18:10.863 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-18T20:18:11.095 DEBUG:teuthology.parallel:result is None 2024-03-18T20:18:11.397 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-18T20:18:11.397 INFO:teuthology.orchestra.run.smithi169.stdout:Installed products updated. 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: automake-1.16.1-8.el8.noarch 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: bison-3.0.4-10.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: flex-2.6.1-9.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-18T20:18:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-7.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libpq-13.11-1.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-18T20:18:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-18T20:18:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-18T20:18:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-18T20:18:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-18T20:18:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-2.6-1.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-18T20:18:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-18T20:18:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-18T20:18:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-18T20:18:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-18T20:18:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:18:12.753 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-18T20:18:13.616 DEBUG:teuthology.parallel:result is None 2024-03-18T20:18:13.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:13.840 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T20:18:13.868 INFO:teuthology.orchestra.run.smithi098.stdout:18.2.2-914.g49bdc511.el8 2024-03-18T20:18:13.869 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-914.g49bdc511.el8 2024-03-18T20:18:13.869 INFO:teuthology.task.install:The correct ceph version 18.2.2-914.g49bdc511 is installed. 2024-03-18T20:18:13.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:14.012 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T20:18:14.041 INFO:teuthology.orchestra.run.smithi136.stdout:18.2.2-914.g49bdc511.el8 2024-03-18T20:18:14.041 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-914.g49bdc511.el8 2024-03-18T20:18:14.042 INFO:teuthology.task.install:The correct ceph version 18.2.2-914.g49bdc511 is installed. 2024-03-18T20:18:14.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:14.260 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T20:18:14.288 INFO:teuthology.orchestra.run.smithi169.stdout:18.2.2-914.g49bdc511.el8 2024-03-18T20:18:14.288 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-914.g49bdc511.el8 2024-03-18T20:18:14.288 INFO:teuthology.task.install:The correct ceph version 18.2.2-914.g49bdc511 is installed. 2024-03-18T20:18:14.290 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T20:18:14.290 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:18:14.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T20:18:14.340 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:18:14.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T20:18:14.389 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:18:14.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T20:18:14.439 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T20:18:14.439 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:18:14.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T20:18:14.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T20:18:14.542 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:18:14.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T20:18:14.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T20:18:14.645 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:18:14.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T20:18:14.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T20:18:14.749 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T20:18:14.749 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:18:14.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T20:18:14.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T20:18:14.853 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:18:14.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T20:18:14.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T20:18:14.956 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:18:14.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T20:18:14.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T20:18:15.056 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T20:18:15.057 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:18:15.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T20:18:15.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T20:18:15.159 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:18:15.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T20:18:15.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T20:18:15.262 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:18:15.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T20:18:15.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T20:18:15.361 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T20:18:15.442 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-18T20:18:15.442 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:15.442 INFO:tasks.cephadm:Cluster fsid is a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:15.443 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T20:18:15.443 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.136', 'mon.c': '172.21.15.169'} 2024-03-18T20:18:15.443 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-03-18T20:18:15.443 INFO:tasks.cephadm:First mgr is x 2024-03-18T20:18:15.443 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T20:18:15.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-18T20:18:15.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-03-18T20:18:15.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-03-18T20:18:15.540 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T20:18:15.541 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:15.681 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/77701/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-914-g49bdc511', 'node_name': '172.21.4.64+confusa02', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-914.g49bdc511'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 02:21:43.411752', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri10-testing-2024-03-15-1653-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-03-18T20:18:15.898 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri10-testing-2024-03-15-1653-reef, sha1 49bdc51142759622a281d3e5327eebd07cbb8bb3 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:18:15.900 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/x86_64/flavors/default/cephadm 2024-03-18T20:18:15.900 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:16.885 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-rw-r--. 1 ubuntu ubuntu 702420 Mar 18 20:18 /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:16.886 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:17.689 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-rw-r--. 1 ubuntu ubuntu 702420 Mar 18 20:18 /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:17.690 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:18.107 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-rw-r--. 1 ubuntu ubuntu 702420 Mar 18 20:18 /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:18.108 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:18.129 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:18.151 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T20:18:18.201 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 on all hosts... 2024-03-18T20:18:18.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 pull 2024-03-18T20:18:18.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 pull 2024-03-18T20:18:18.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 pull 2024-03-18T20:18:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3... 2024-03-18T20:18:19.686 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3... 2024-03-18T20:18:19.974 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3... 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "015bdba7dcef68659e3352b62f3bbf78033bb173f439b9af177f23d5c5943567", 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1b5dd227b04788a41794658b6571950ee4d311fa0172206cd8a725de73702aee" 2024-03-18T20:18:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:18:43.617 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:18:45.104 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-18T20:18:45.104 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:18:45.104 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "015bdba7dcef68659e3352b62f3bbf78033bb173f439b9af177f23d5c5943567", 2024-03-18T20:18:45.105 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-03-18T20:18:45.105 INFO:teuthology.orchestra.run.smithi169.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1b5dd227b04788a41794658b6571950ee4d311fa0172206cd8a725de73702aee" 2024-03-18T20:18:45.105 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-18T20:18:45.105 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "015bdba7dcef68659e3352b62f3bbf78033bb173f439b9af177f23d5c5943567", 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1b5dd227b04788a41794658b6571950ee4d311fa0172206cd8a725de73702aee" 2024-03-18T20:18:49.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-03-18T20:18:49.841 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-18T20:18:49.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-18T20:18:49.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-03-18T20:18:49.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-03-18T20:18:49.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-18T20:18:49.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-03-18T20:18:50.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-03-18T20:18:50.054 INFO:tasks.cephadm:Writing seed config... 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T20:18:50.055 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T20:18:50.056 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T20:18:50.057 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T20:18:50.058 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T20:18:50.058 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T20:18:50.058 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-18T20:18:50.058 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:18:50.059 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T20:18:50.078 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a6e2b072-e564-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T20:18:50.081 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T20:18:50.121 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T20:18:50.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:18:50.151 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T20:18:50.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 -v bootstrap --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:18:50.179 INFO:journalctl@ceph.mgr.x.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:18:50.314 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-18T20:18:50.314 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3', '-v', 'bootstrap', '--fsid', 'a6e2b072-e564-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-03-18T20:18:50.343 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.6.1 2024-03-18T20:18:50.344 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T20:18:50.344 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-03-18T20:18:50.375 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.6.1 2024-03-18T20:18:50.375 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-03-18T20:18:50.375 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-03-18T20:18:50.381 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T20:18:50.382 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T20:18:50.387 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T20:18:50.387 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:18:50.393 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-03-18T20:18:50.399 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-03-18T20:18:50.399 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-03-18T20:18:50.399 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-03-18T20:18:50.427 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.6.1 2024-03-18T20:18:50.427 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.6.1 is present 2024-03-18T20:18:50.427 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-03-18T20:18:50.427 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-03-18T20:18:50.433 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T20:18:50.433 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T20:18:50.438 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T20:18:50.438 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:18:50.444 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-03-18T20:18:50.450 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-03-18T20:18:50.450 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-03-18T20:18:50.450 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 139774744521192 on /run/cephadm/a6e2b072-e564-11ee-95c9-87774f69a715.lock 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 139774744521192 acquired on /run/cephadm/a6e2b072-e564-11ee-95c9-87774f69a715.lock 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-03-18T20:18:50.451 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-18T20:18:50.455 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-03-18T20:18:50.455 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-03-18T20:18:50.458 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T20:18:50.458 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-03-18T20:18:50.458 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-03-18T20:18:50.460 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T20:18:50.460 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::6960:dbbe:1345:fedd/64 scope link noprefixroute 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-03-18T20:18:50.461 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-03-18T20:18:50.462 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T20:18:50.462 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T20:18:50.463 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3... 2024-03-18T20:18:50.836 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 015bdba7dcef68659e3352b62f3bbf78033bb173f439b9af177f23d5c5943567 2024-03-18T20:18:50.836 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3... 2024-03-18T20:18:50.837 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T20:18:50.837 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:4bbf42c993c119ce89c30a67b341883c7867a88bea0dc325a65e3df8d508a4ef 2024-03-18T20:18:50.837 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T20:18:50.837 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:015bdba7dcef68659e3352b62f3bbf78033bb173f439b9af177f23d5c5943567 2024-03-18T20:18:50.837 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T20:18:51.530 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable) 2024-03-18T20:18:51.530 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable) 2024-03-18T20:18:51.530 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T20:18:52.102 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-03-18T20:18:52.102 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-03-18T20:18:52.686 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQCsofhlO1/jExAAJb6vDUuM+fy7E+GZ/UayVA== 2024-03-18T20:18:53.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQCsofhl6JFBNhAAvxhJVex+X3qmWUlHtVLRnA== 2024-03-18T20:18:53.869 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQCtofhl+u9MHxAAYKh22Y0WEQ8Bocs+oc901Q== 2024-03-18T20:18:53.869 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = pacific 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:54.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T20:18:54.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:18:54.491 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-03-18T20:18:55.092 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-03-18T20:18:55.218 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-18T20:18:55.309 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T20:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target. 2024-03-18T20:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target. 2024-03-18T20:18:56.021 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a 2024-03-18T20:18:56.022 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T20:18:56.109 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target.wants/ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715@.service. 2024-03-18T20:18:57.766 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T20:18:57.766 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-18T20:18:57.775 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T20:18:57.775 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:18:57.775 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-18T20:18:57.776 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T20:18:57.776 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-03-18T20:18:57.776 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-03-18T20:18:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:57 smithi098 ceph-mon[42976]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.477776s) 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.163+0000 7f2a6acd9700 1 Processor -- start 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.163+0000 7f2a6acd9700 1 -- start start 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.163+0000 7f2a6acd9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.163+0000 7f2a6acd9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64105ca0 con 0x7f2a641052b0 2024-03-18T20:18:58.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a68a75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a68a75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42894/0 (socket says 172.21.15.98:42894) 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a68a75700 1 -- 172.21.15.98:0/1914459884 learned_addr learned my addr 172.21.15.98:0/1914459884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a68a75700 1 -- 172.21.15.98:0/1914459884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a641064b0 con 0x7f2a641052b0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a68a75700 1 --2- 172.21.15.98:0/1914459884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a580098a0 tx=0x7f2a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92bb8c8c635c4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a637fe700 1 -- 172.21.15.98:0/1914459884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a58004160 con 0x7f2a641052b0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a637fe700 1 -- 172.21.15.98:0/1914459884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a580042c0 con 0x7f2a641052b0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.164+0000 7f2a637fe700 1 -- 172.21.15.98:0/1914459884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a580045a0 con 0x7f2a641052b0 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 -- 172.21.15.98:0/1914459884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 msgr2=0x7f2a641056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 --2- 172.21.15.98:0/1914459884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a580098a0 tx=0x7f2a58009bb0 comp rx=0 tx=0).stop 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 -- 172.21.15.98:0/1914459884 shutdown_connections 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 --2- 172.21.15.98:0/1914459884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a641056d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 -- 172.21.15.98:0/1914459884 >> 172.21.15.98:0/1914459884 conn(0x7f2a641008b0 msgr2=0x7f2a64102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:58.605 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 -- 172.21.15.98:0/1914459884 shutdown_connections 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 -- 172.21.15.98:0/1914459884 wait complete. 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.165+0000 7f2a6acd9700 1 Processor -- start 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a6acd9700 1 -- start start 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a6acd9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a6acd9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6418e360 con 0x7f2a641052b0 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a68a75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a68a75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42904/0 (socket says 172.21.15.98:42904) 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a68a75700 1 -- 172.21.15.98:0/445451624 learned_addr learned my addr 172.21.15.98:0/445451624 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a68a75700 1 -- 172.21.15.98:0/445451624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a58009550 con 0x7f2a641052b0 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a68a75700 1 --2- 172.21.15.98:0/445451624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a58004000 tx=0x7f2a58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a58011630 con 0x7f2a641052b0 2024-03-18T20:18:58.606 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a58011790 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a58011a30 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.166+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6418c5a0 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.167+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6418ca40 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.167+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2a58022070 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.167+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2a5801c450 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.168+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6404f2d0 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.170+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2a5801b450 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.239+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2a641934b0 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.239+0000 7f2a61ffb700 1 -- 172.21.15.98:0/445451624 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f2a5801b5f0 con 0x7f2a641052b0 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 msgr2=0x7f2a6418de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 --2- 172.21.15.98:0/445451624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2a58004000 tx=0x7f2a58004750 comp rx=0 tx=0).stop 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 shutdown_connections 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 --2- 172.21.15.98:0/445451624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a641052b0 0x7f2a6418de20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:58.607 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 >> 172.21.15.98:0/445451624 conn(0x7f2a641008b0 msgr2=0x7f2a641906d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:58.608 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 shutdown_connections 2024-03-18T20:18:58.608 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.240+0000 7f2a6acd9700 1 -- 172.21.15.98:0/445451624 wait complete. 2024-03-18T20:18:58.608 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-03-18T20:18:58.608 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T20:18:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:18:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:18:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: fsmap 2024-03-18T20:18:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T20:18:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: mgrmap e1: no daemons active 2024-03-18T20:18:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:58 smithi098 ceph-mon[42976]: from='client.? 172.21.15.98:0/445451624' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T20:18:59.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:59.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T20:18:59.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:18:59.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T20:18:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T20:18:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.936+0000 7f914d32e700 1 Processor -- start 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.936+0000 7f914d32e700 1 -- start start 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f914d32e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f914d32e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9148105120 con 0x7f9148104730 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42910/0 (socket says 172.21.15.98:42910) 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9146ffd700 1 -- 172.21.15.98:0/1357235315 learned_addr learned my addr 172.21.15.98:0/1357235315 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9146ffd700 1 -- 172.21.15.98:0/1357235315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91481076c0 con 0x7f9148104730 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9146ffd700 1 --2- 172.21.15.98:0/1357235315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f91380098a0 tx=0x7f9138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a7ed477cfb51f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9145ffb700 1 -- 172.21.15.98:0/1357235315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9138004160 con 0x7f9148104730 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9145ffb700 1 -- 172.21.15.98:0/1357235315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91380042c0 con 0x7f9148104730 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.937+0000 7f9145ffb700 1 -- 172.21.15.98:0/1357235315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91380045a0 con 0x7f9148104730 2024-03-18T20:18:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 -- 172.21.15.98:0/1357235315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 msgr2=0x7f9148104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 --2- 172.21.15.98:0/1357235315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f91380098a0 tx=0x7f9138009bb0 comp rx=0 tx=0).stop 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 -- 172.21.15.98:0/1357235315 shutdown_connections 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 --2- 172.21.15.98:0/1357235315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148104b50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 -- 172.21.15.98:0/1357235315 >> 172.21.15.98:0/1357235315 conn(0x7f91481000e0 msgr2=0x7f9148102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 -- 172.21.15.98:0/1357235315 shutdown_connections 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 -- 172.21.15.98:0/1357235315 wait complete. 2024-03-18T20:18:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.938+0000 7f914d32e700 1 Processor -- start 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f914d32e700 1 -- start start 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f914d32e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f914d32e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9148107430 con 0x7f9148104730 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42924/0 (socket says 172.21.15.98:42924) 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f9146ffd700 1 -- 172.21.15.98:0/3713959363 learned_addr learned my addr 172.21.15.98:0/3713959363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f9146ffd700 1 -- 172.21.15.98:0/3713959363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138009550 con 0x7f9148104730 2024-03-18T20:18:59.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f9146ffd700 1 --2- 172.21.15.98:0/3713959363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9138004000 tx=0x7f9138004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9138011630 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.939+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9138011790 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.940+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9148105670 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.940+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9138011a30 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.940+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9148105b10 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.941+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9138021070 con 0x7f9148104730 2024-03-18T20:18:59.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.941+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f913801b780 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.941+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914804f2d0 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:58.943+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f913801acc0 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.008+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9148192fd0 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.012+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f913801ae60 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.012+0000 7f912ffff700 1 -- 172.21.15.98:0/3713959363 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f913802c5a0 con 0x7f9148104730 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 msgr2=0x7f9148106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:59.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 --2- 172.21.15.98:0/3713959363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9138004000 tx=0x7f9138004750 comp rx=0 tx=0).stop 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 shutdown_connections 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 --2- 172.21.15.98:0/3713959363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9148104730 0x7f9148106ef0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 >> 172.21.15.98:0/3713959363 conn(0x7f91481000e0 msgr2=0x7f9148190730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 shutdown_connections 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.013+0000 7f914d32e700 1 -- 172.21.15.98:0/3713959363 wait complete. 2024-03-18T20:18:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-03-18T20:19:00.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:59 smithi098 ceph-mon[42976]: from='client.? 172.21.15.98:0/3713959363' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T20:19:00.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:18:59 smithi098 ceph-mon[42976]: from='client.? 172.21.15.98:0/3713959363' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T20:19:00.157 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc6576ff700 1 Processor -- start 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc6576ff700 1 -- start start 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc6576ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc6576ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650079e50 con 0x7fc650106e60 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc65549b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc65549b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42938/0 (socket says 172.21.15.98:42938) 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc65549b700 1 -- 172.21.15.98:0/772233211 learned_addr learned my addr 172.21.15.98:0/772233211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc65549b700 1 -- 172.21.15.98:0/772233211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650109790 con 0x7fc650106e60 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.710+0000 7fc65549b700 1 --2- 172.21.15.98:0/772233211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc6440098a0 tx=0x7fc644009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0322a567a67a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc643fff700 1 -- 172.21.15.98:0/772233211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc644004160 con 0x7fc650106e60 2024-03-18T20:19:00.158 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc643fff700 1 -- 172.21.15.98:0/772233211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc6440042c0 con 0x7fc650106e60 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc643fff700 1 -- 172.21.15.98:0/772233211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc644011940 con 0x7fc650106e60 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 -- 172.21.15.98:0/772233211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 msgr2=0x7fc650109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 --2- 172.21.15.98:0/772233211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc6440098a0 tx=0x7fc644009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 -- 172.21.15.98:0/772233211 shutdown_connections 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 --2- 172.21.15.98:0/772233211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650109250 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 -- 172.21.15.98:0/772233211 >> 172.21.15.98:0/772233211 conn(0x7fc650100870 msgr2=0x7fc650102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 -- 172.21.15.98:0/772233211 shutdown_connections 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.711+0000 7fc6576ff700 1 -- 172.21.15.98:0/772233211 wait complete. 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc6576ff700 1 Processor -- start 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc6576ff700 1 -- start start 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc6576ff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc6576ff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650106bd0 con 0x7fc650106e60 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc65549b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:00.159 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc65549b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42954/0 (socket says 172.21.15.98:42954) 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc65549b700 1 -- 172.21.15.98:0/3594303826 learned_addr learned my addr 172.21.15.98:0/3594303826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.712+0000 7fc65549b700 1 -- 172.21.15.98:0/3594303826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644009550 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc65549b700 1 --2- 172.21.15.98:0/3594303826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc644006af0 tx=0x7fc644011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc644003bf0 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc64401b8b0 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc650104e10 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc644023800 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6501052b0 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc64402c3f0 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.713+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc644019710 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.714+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65010a4d0 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.716+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fc64401bd70 con 0x7fc650106e60 2024-03-18T20:19:00.160 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.783+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc650105910 con 0x7fc650106e60 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6427fc700 1 -- 172.21.15.98:0/3594303826 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fc64401bad0 con 0x7fc650106e60 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 msgr2=0x7fc650106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 --2- 172.21.15.98:0/3594303826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc644006af0 tx=0x7fc644011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 shutdown_connections 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 --2- 172.21.15.98:0/3594303826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc650106e60 0x7fc650106690 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 >> 172.21.15.98:0/3594303826 conn(0x7fc650100870 msgr2=0x7fc6501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 shutdown_connections 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:18:59.784+0000 7fc6576ff700 1 -- 172.21.15.98:0/3594303826 wait complete. 2024-03-18T20:19:00.161 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-03-18T20:19:00.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 systemd[1]: Stopping Ceph mon.a for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:19:00.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a[42953]: 2024-03-18T20:19:00.351+0000 7f8b7d9bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:19:00.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a[42953]: 2024-03-18T20:19:00.351+0000 7f8b7d9bb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T20:19:00.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 bash[43303]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a 2024-03-18T20:19:01.239 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 systemd[1]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service: Succeeded. 2024-03-18T20:19:01.239 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 systemd[1]: Stopped Ceph mon.a for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:19:01.239 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:00 smithi098 systemd[1]: Starting Ceph mon.a for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:19:01.313 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T20:19:01.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 bash[43424]: dd974b9940e385bb9bc723f7bf133365c20b9bd0b66dcacce4cb476aec7b68ec 2024-03-18T20:19:01.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 systemd[1]: Started Ceph mon.a for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:19:01.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T20:19:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable), process ceph-mon, pid 7 2024-03-18T20:19:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: pidfile_write: ignore empty --pid-file 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: load: jerasure load: lrc 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: RocksDB version: 7.9.2 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Git sha 0 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Compile date 2024-03-16 00:14:49 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: DB SUMMARY 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: DB Session ID: TPWGK0WHHVFNEY2VBOZP 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: CURRENT file: CURRENT 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T20:19:01.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97008 ; 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.error_if_exists: 0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.create_if_missing: 0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.paranoid_checks: 1 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.env: 0x557b3ecdb720 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.info_log: 0x557b40f974e0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.statistics: (nil) 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.use_fsync: 0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_log_file_size: 0 2024-03-18T20:19:01.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_fallocate: 1 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.use_direct_reads: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.db_log_dir: 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.wal_dir: 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T20:19:01.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.write_buffer_manager: 0x557b40230320 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T20:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.unordered_write: 0 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.row_cache: None 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.wal_filter: None 2024-03-18T20:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.two_write_queues: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.wal_compression: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.atomic_flush: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.log_readahead_size: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T20:19:01.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T20:19:01.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T20:19:01.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T20:19:01.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_background_jobs: 2 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_background_compactions: -1 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_subcompactions: 1 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T20:19:01.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_open_files: -1 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_background_flushes: -1 2024-03-18T20:19:01.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Compression algorithms supported: 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kZSTD supported: 0 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kXpressCompression supported: 0 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kZlibCompression supported: 1 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kSnappyCompression supported: 1 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kLZ4Compression supported: 1 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: kBZip2Compression supported: 0 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T20:19:01.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.merge_operator: 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_filter: None 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557b40f975e0) 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T20:19:01.604 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x557b402b11f0 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-03-18T20:19:01.605 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-03-18T20:19:01.606 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression: NoCompression 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.num_levels: 7 2024-03-18T20:19:01.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T20:19:01.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T20:19:01.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T20:19:01.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T20:19:01.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T20:19:01.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.table_properties_collectors: 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.inplace_update_support: 0 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T20:19:01.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.bloom_locality: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.max_successive_merges: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T20:19:01.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.ttl: 2592000 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enable_blob_files: false 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.min_blob_size: 0 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T20:19:01.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 610fce0a-e2b1-49dc-b9dd-3d4c7c0bfe1a 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793141324560, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793141325643, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92486, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 305, "table_properties": {"data_size": 90491, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13481, "raw_average_key_size": 50, "raw_value_size": 83115, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 268, "num_filter_entries": 268, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710793141, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "610fce0a-e2b1-49dc-b9dd-3d4c7c0bfe1a", "db_session_id": "TPWGK0WHHVFNEY2VBOZP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793141325734, "job": 1, "event": "recovery_finished"} 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T20:19:01.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557b4034e000 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: DB pointer 0x557b4033c000 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-18T20:19:01.617 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 92.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 92.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T20:19:01.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 26.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 26.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x557b402b11f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-18T20:19:01.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???) e1 preinit fsid a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).mds e1 new map 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).mds e1 print_map 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-03-18T20:19:01.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: fsmap 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T20:19:01.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:01 smithi098 ceph-mon[43462]: mgrmap e1: no daemons active 2024-03-18T20:19:02.104 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbef26700 1 Processor -- start 2024-03-18T20:19:02.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbef26700 1 -- start start 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbef26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbef26700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8105ca0 con 0x7f6bb81052b0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbccc2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.646+0000 7f6bbccc2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42962/0 (socket says 172.21.15.98:42962) 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bbccc2700 1 -- 172.21.15.98:0/3447799260 learned_addr learned my addr 172.21.15.98:0/3447799260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bbccc2700 1 -- 172.21.15.98:0/3447799260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb81064b0 con 0x7f6bb81052b0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bbccc2700 1 --2- 172.21.15.98:0/3447799260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80098a0 tx=0x7f6ba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8226e22284f75685 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bb77fe700 1 -- 172.21.15.98:0/3447799260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ba8004160 con 0x7f6bb81052b0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bb77fe700 1 -- 172.21.15.98:0/3447799260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6ba80042c0 con 0x7f6bb81052b0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.647+0000 7f6bb77fe700 1 -- 172.21.15.98:0/3447799260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ba8011940 con 0x7f6bb81052b0 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- 172.21.15.98:0/3447799260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 msgr2=0x7f6bb81056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 --2- 172.21.15.98:0/3447799260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80098a0 tx=0x7f6ba8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- 172.21.15.98:0/3447799260 shutdown_connections 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 --2- 172.21.15.98:0/3447799260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb81056d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- 172.21.15.98:0/3447799260 >> 172.21.15.98:0/3447799260 conn(0x7f6bb81008f0 msgr2=0x7f6bb8102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- 172.21.15.98:0/3447799260 shutdown_connections 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- 172.21.15.98:0/3447799260 wait complete. 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 Processor -- start 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.648+0000 7f6bbef26700 1 -- start start 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbef26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbef26700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb819a5b0 con 0x7f6bb81052b0 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbccc2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbccc2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42970/0 (socket says 172.21.15.98:42970) 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbccc2700 1 -- 172.21.15.98:0/2443025996 learned_addr learned my addr 172.21.15.98:0/2443025996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:02.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbccc2700 1 -- 172.21.15.98:0/2443025996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8009550 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbccc2700 1 --2- 172.21.15.98:0/2443025996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8003f20 tx=0x7f6ba8011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ba80036a0 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6ba8003c70 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ba801be80 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb819a7b0 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.649+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb819ac50 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.650+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6ba8023790 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.650+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ba802cd50 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.650+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb804f2d0 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.653+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f6ba801b440 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.718+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6bb81096e0 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bb5ffb700 1 -- 172.21.15.98:0/2443025996 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6ba8006d60 con 0x7f6bb81052b0 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 msgr2=0x7f6bb819a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:02.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 --2- 172.21.15.98:0/2443025996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8003f20 tx=0x7f6ba8011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 shutdown_connections 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 --2- 172.21.15.98:0/2443025996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81052b0 0x7f6bb819a070 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 >> 172.21.15.98:0/2443025996 conn(0x7f6bb81008f0 msgr2=0x7f6bb8101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.724+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 shutdown_connections 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:01.725+0000 7f6bbef26700 1 -- 172.21.15.98:0/2443025996 wait complete. 2024-03-18T20:19:02.112 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T20:19:02.132 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:19:02.132 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-03-18T20:19:02.133 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T20:19:02.133 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T20:19:02.339 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x 2024-03-18T20:19:02.339 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T20:19:02.434 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715.target.wants/ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a6e2b072-e564-11ee-95c9-87774f69a715@.service. 2024-03-18T20:19:02.644 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:02 smithi098 systemd[1]: Starting Ceph mgr.x for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:19:02.839 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T20:19:02.839 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-18T20:19:02.846 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T20:19:02.846 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:19:02.847 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-18T20:19:02.847 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T20:19:02.855 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T20:19:02.855 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-18T20:19:02.862 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T20:19:02.862 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:19:02.862 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-18T20:19:02.862 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T20:19:02.863 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-03-18T20:19:02.863 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-03-18T20:19:02.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:02 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2443025996' entity='client.admin' 2024-03-18T20:19:02.962 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:02 smithi098 bash[43699]: d79e528d7c557b0607bc6d6127d4b9dd2e04f870873eb2c44b6c1a68e4c1e839 2024-03-18T20:19:02.963 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:02 smithi098 systemd[1]: Started Ceph mgr.x for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:19:03.509 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:03.240+0000 7f0acdbbd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:19:03.509 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:03.337+0000 7f0acdbbd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:19:03.509 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:03.427+0000 7f0acdbbd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T20:19:03.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 1, 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T20:19:03.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T20:19:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T20:18:57.764467+0000", 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.235+0000 7f6afbc7e700 1 Processor -- start 2024-03-18T20:19:03.710 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.235+0000 7f6afbc7e700 1 -- start start 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.235+0000 7f6afbc7e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.235+0000 7f6afbc7e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4079e10 con 0x7f6af4104c90 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af9a1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af9a1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36914/0 (socket says 172.21.15.98:36914) 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af9a1a700 1 -- 172.21.15.98:0/358115834 learned_addr learned my addr 172.21.15.98:0/358115834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af9a1a700 1 -- 172.21.15.98:0/358115834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af41055f0 con 0x7f6af4104c90 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af9a1a700 1 --2- 172.21.15.98:0/358115834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6af00098a0 tx=0x7f6af0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b643b98757903cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.236+0000 7f6af8a18700 1 -- 172.21.15.98:0/358115834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6af0004160 con 0x7f6af4104c90 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6af8a18700 1 -- 172.21.15.98:0/358115834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6af00042c0 con 0x7f6af4104c90 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6af8a18700 1 -- 172.21.15.98:0/358115834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6af0011940 con 0x7f6af4104c90 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 -- 172.21.15.98:0/358115834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 msgr2=0x7f6af41050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 --2- 172.21.15.98:0/358115834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6af00098a0 tx=0x7f6af0009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 -- 172.21.15.98:0/358115834 shutdown_connections 2024-03-18T20:19:03.711 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 --2- 172.21.15.98:0/358115834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af41050b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 -- 172.21.15.98:0/358115834 >> 172.21.15.98:0/358115834 conn(0x7f6af4100870 msgr2=0x7f6af4102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 -- 172.21.15.98:0/358115834 shutdown_connections 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.237+0000 7f6afbc7e700 1 -- 172.21.15.98:0/358115834 wait complete. 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6afbc7e700 1 Processor -- start 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6afbc7e700 1 -- start start 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6afbc7e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6afbc7e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af419c7b0 con 0x7f6af4104c90 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6af9a1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6af9a1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36918/0 (socket says 172.21.15.98:36918) 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.238+0000 7f6af9a1a700 1 -- 172.21.15.98:0/1629574689 learned_addr learned my addr 172.21.15.98:0/1629574689 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6af9a1a700 1 -- 172.21.15.98:0/1629574689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009550 con 0x7f6af4104c90 2024-03-18T20:19:03.712 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6af9a1a700 1 --2- 172.21.15.98:0/1629574689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6af0003f20 tx=0x7f6af0011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6af00037d0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6af001b440 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af419c9b0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6af0023590 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af419ce50 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6af0023b50 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.239+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6af00193d0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.240+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af4195fc0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.242+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f6af001b5b0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.314+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6af402ccb0 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.314+0000 7f6aeaffd700 1 -- 172.21.15.98:0/1629574689 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6af001b750 con 0x7f6af4104c90 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 msgr2=0x7f6af419c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:03.713 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 --2- 172.21.15.98:0/1629574689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6af0003f20 tx=0x7f6af0011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 shutdown_connections 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 --2- 172.21.15.98:0/1629574689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6af4104c90 0x7f6af419c270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 >> 172.21.15.98:0/1629574689 conn(0x7f6af4100870 msgr2=0x7f6af4193100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 shutdown_connections 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:03.315+0000 7f6afbc7e700 1 -- 172.21.15.98:0/1629574689 wait complete. 2024-03-18T20:19:03.714 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-03-18T20:19:03.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1629574689' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:19:03.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:03.512+0000 7f0acdbbd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:19:04.485 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.150+0000 7f0acdbbd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:04.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.398+0000 7f0acdbbd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:19:04.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.481+0000 7f0acdbbd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:19:04.748 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.589+0000 7f0acdbbd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:05.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.744+0000 7f0acdbbd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:19:05.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.869+0000 7f0acdbbd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:19:05.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:04.953+0000 7f0acdbbd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:19:05.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:05.318+0000 7f0acdbbd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:19:06.021 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:05.696+0000 7f0acdbbd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:19:06.021 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:05.920+0000 7f0acdbbd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:19:06.021 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:06.017+0000 7f0acdbbd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:19:06.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:06 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2533342490' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T20:19:06.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.556 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T20:19:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.558 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T20:19:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T20:18:57.764467+0000", 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.049+0000 7fddd5a42700 1 Processor -- start 2024-03-18T20:19:06.560 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.049+0000 7fddd5a42700 1 -- start start 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.049+0000 7fddd5a42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.049+0000 7fddd5a42700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd0079e90 con 0x7fddd0106e20 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36924/0 (socket says 172.21.15.98:36924) 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddceffd700 1 -- 172.21.15.98:0/192648959 learned_addr learned my addr 172.21.15.98:0/192648959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddceffd700 1 -- 172.21.15.98:0/192648959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd0109750 con 0x7fddd0106e20 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddceffd700 1 --2- 172.21.15.98:0/192648959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddc00098a0 tx=0x7fddc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2123e33af1e0f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddcdffb700 1 -- 172.21.15.98:0/192648959 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddc0004160 con 0x7fddd0106e20 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddcdffb700 1 -- 172.21.15.98:0/192648959 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fddc00042c0 con 0x7fddd0106e20 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.050+0000 7fddcdffb700 1 -- 172.21.15.98:0/192648959 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddc0011940 con 0x7fddd0106e20 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 -- 172.21.15.98:0/192648959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 msgr2=0x7fddd0109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 --2- 172.21.15.98:0/192648959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddc00098a0 tx=0x7fddc0009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 -- 172.21.15.98:0/192648959 shutdown_connections 2024-03-18T20:19:06.561 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 --2- 172.21.15.98:0/192648959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd0109210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 -- 172.21.15.98:0/192648959 >> 172.21.15.98:0/192648959 conn(0x7fddd0100870 msgr2=0x7fddd0102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 -- 172.21.15.98:0/192648959 shutdown_connections 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 -- 172.21.15.98:0/192648959 wait complete. 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.051+0000 7fddd5a42700 1 Processor -- start 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddd5a42700 1 -- start start 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddd5a42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddd5a42700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd019a570 con 0x7fddd0106e20 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36932/0 (socket says 172.21.15.98:36932) 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddceffd700 1 -- 172.21.15.98:0/2533342490 learned_addr learned my addr 172.21.15.98:0/2533342490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddceffd700 1 -- 172.21.15.98:0/2533342490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0009550 con 0x7fddd0106e20 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddceffd700 1 --2- 172.21.15.98:0/2533342490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fddc0003f20 tx=0x7fddc0011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:06.562 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.052+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddc00037d0 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fddc001b440 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fddc0023590 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd019a770 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd019ac10 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fddc0023b50 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fddc00193d0 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.053+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd010a4f0 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.056+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fddc002ce50 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.126+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fddd010a6b0 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.127+0000 7fddd4a40700 1 -- 172.21.15.98:0/2533342490 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fddc0006d60 con 0x7fddd0106e20 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 msgr2=0x7fddd019a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 --2- 172.21.15.98:0/2533342490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fddc0003f20 tx=0x7fddc0011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 shutdown_connections 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 --2- 172.21.15.98:0/2533342490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddd0106e20 0x7fddd019a030 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:06.563 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 >> 172.21.15.98:0/2533342490 conn(0x7fddd0100870 msgr2=0x7fddd01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:06.564 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 shutdown_connections 2024-03-18T20:19:06.564 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:06.128+0000 7fddd5a42700 1 -- 172.21.15.98:0/2533342490 wait complete. 2024-03-18T20:19:06.564 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-03-18T20:19:07.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:06.794+0000 7f0acdbbd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:19:07.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:06.890+0000 7f0acdbbd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:19:07.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:07.036+0000 7f0acdbbd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:19:08.028 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:07.651+0000 7f0acdbbd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:19:08.028 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:07.830+0000 7f0acdbbd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:19:08.342 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:08.024+0000 7f0acdbbd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:19:09.238 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:08.953+0000 7f0acdbbd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:19:09.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:09.234+0000 7f0acdbbd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:09.505 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:09.325+0000 7f0acdbbd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:09.519+0000 7f0acdbbd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: Activating manager daemon x 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: mgrmap e2: x(active, starting, since 0.00302998s) 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:19:09.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: Manager daemon x is now available 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:19:09.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:09 smithi098 ceph-mon[43462]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:19:10.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:10 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2759934608' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:19:10.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:10 smithi098 ceph-mon[43462]: mgrmap e3: x(active, since 1.00713s) 2024-03-18T20:19:11.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T20:19:11.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T20:19:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T20:19:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T20:19:11.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T20:18:57.764467+0000", 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f6a01767700 1 Processor -- start 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f6a01767700 1 -- start start 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f6a01767700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f6a01767700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc107400 con 0x7f69fc106a10 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f69faffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:11.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f69faffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37012/0 (socket says 172.21.15.98:37012) 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f69faffd700 1 -- 172.21.15.98:0/376029536 learned_addr learned my addr 172.21.15.98:0/376029536 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f69faffd700 1 -- 172.21.15.98:0/376029536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc104f50 con 0x7f69fc106a10 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.941+0000 7f69faffd700 1 --2- 172.21.15.98:0/376029536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0098a0 tx=0x7f69ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6401c4ebd080818 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f69f9ffb700 1 -- 172.21.15.98:0/376029536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69ec004160 con 0x7f69fc106a10 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f69f9ffb700 1 -- 172.21.15.98:0/376029536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f69ec0042c0 con 0x7f69fc106a10 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f69f9ffb700 1 -- 172.21.15.98:0/376029536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69ec011940 con 0x7f69fc106a10 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 -- 172.21.15.98:0/376029536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 msgr2=0x7f69fc106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 --2- 172.21.15.98:0/376029536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0098a0 tx=0x7f69ec009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 -- 172.21.15.98:0/376029536 shutdown_connections 2024-03-18T20:19:11.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 --2- 172.21.15.98:0/376029536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc106e30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 -- 172.21.15.98:0/376029536 >> 172.21.15.98:0/376029536 conn(0x7f69fc100950 msgr2=0x7f69fc102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 -- 172.21.15.98:0/376029536 shutdown_connections 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.942+0000 7f6a01767700 1 -- 172.21.15.98:0/376029536 wait complete. 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f6a01767700 1 Processor -- start 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f6a01767700 1 -- start start 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f6a01767700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f6a01767700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc19c890 con 0x7f69fc106a10 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f69faffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f69faffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37016/0 (socket says 172.21.15.98:37016) 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f69faffd700 1 -- 172.21.15.98:0/2759934608 learned_addr learned my addr 172.21.15.98:0/2759934608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f69faffd700 1 -- 172.21.15.98:0/2759934608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec009550 con 0x7f69fc106a10 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.943+0000 7f69faffd700 1 --2- 172.21.15.98:0/2759934608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f69ec006af0 tx=0x7f69ec011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:11.446 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69ec003bf0 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f69ec01b8b0 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc19ca90 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69ec023830 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc19cf30 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.944+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 2) v1 ==== 45289+0+0 (secure 0 0 0) 0x7f69ec023990 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.945+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f69ec04c6f0 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.945+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc1967e0 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:09.947+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f69ec023c40 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.018+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f69fc02d0d0 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.018+0000 7f69e3fff700 1 -- 172.21.15.98:0/2759934608 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f69ec02bb40 con 0x7f69fc106a10 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 msgr2=0x7f69fc19c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 --2- 172.21.15.98:0/2759934608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f69ec006af0 tx=0x7f69ec011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 shutdown_connections 2024-03-18T20:19:11.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 --2- 172.21.15.98:0/2759934608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69fc106a10 0x7f69fc19c350 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:11.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 >> 172.21.15.98:0/2759934608 conn(0x7f69fc100950 msgr2=0x7f69fc1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:11.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 shutdown_connections 2024-03-18T20:19:11.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:10.019+0000 7f6a01767700 1 -- 172.21.15.98:0/2759934608 wait complete. 2024-03-18T20:19:11.448 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (3/15)... 2024-03-18T20:19:13.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:13 smithi098 ceph-mon[43462]: mgrmap e4: x(active, since 2s) 2024-03-18T20:19:15.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:15 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3066743419' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:19:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:19:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T20:19:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T20:19:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T20:19:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T20:19:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T20:19:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T20:19:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T20:19:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T20:19:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T20:19:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T20:19:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T20:18:57.764467+0000", 2024-03-18T20:19:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T20:19:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbccc5d700 1 Processor -- start 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbccc5d700 1 -- start start 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbccc5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbccc5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8079e90 con 0x7fbbc8106e20 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57554/0 (socket says 172.21.15.98:57554) 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.954+0000 7fbbc659c700 1 -- 172.21.15.98:0/1408351873 learned_addr learned my addr 172.21.15.98:0/1408351873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbc659c700 1 -- 172.21.15.98:0/1408351873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc8109750 con 0x7fbbc8106e20 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbc659c700 1 --2- 172.21.15.98:0/1408351873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098a0 tx=0x7fbbb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632b63435e57a855 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbc559a700 1 -- 172.21.15.98:0/1408351873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb8004160 con 0x7fbbc8106e20 2024-03-18T20:19:17.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbc559a700 1 -- 172.21.15.98:0/1408351873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbbb80042c0 con 0x7fbbc8106e20 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbc559a700 1 -- 172.21.15.98:0/1408351873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb8011940 con 0x7fbbc8106e20 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 -- 172.21.15.98:0/1408351873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 msgr2=0x7fbbc8109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/1408351873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098a0 tx=0x7fbbb8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 -- 172.21.15.98:0/1408351873 shutdown_connections 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/1408351873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc8109210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 -- 172.21.15.98:0/1408351873 >> 172.21.15.98:0/1408351873 conn(0x7fbbc8100870 msgr2=0x7fbbc8102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 -- 172.21.15.98:0/1408351873 shutdown_connections 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.955+0000 7fbbccc5d700 1 -- 172.21.15.98:0/1408351873 wait complete. 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.956+0000 7fbbccc5d700 1 Processor -- start 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.956+0000 7fbbccc5d700 1 -- start start 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.956+0000 7fbbccc5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.956+0000 7fbbccc5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc819a570 con 0x7fbbc8106e20 2024-03-18T20:19:17.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.956+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57558/0 (socket says 172.21.15.98:57558) 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbc659c700 1 -- 172.21.15.98:0/3066743419 learned_addr learned my addr 172.21.15.98:0/3066743419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbc659c700 1 -- 172.21.15.98:0/3066743419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009550 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbc659c700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8006af0 tx=0x7fbbb8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb8003bf0 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbbb801b8b0 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc819a770 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb8023800 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.957+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc819ac10 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.958+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7fbbb8023a70 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.958+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc810a4f0 con 0x7fbbc8106e20 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.958+0000 7fbbb77fe700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 0x7fbbb003ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:17.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.958+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbbb804cb20 con 0x7fbbc8106e20 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.958+0000 7fbbc5d9b700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 0x7fbbb003ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.959+0000 7fbbc5d9b700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 0x7fbbb003ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006fd0 tx=0x7fbbbc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:14.963+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbbb8023d20 con 0x7fbbc8106e20 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.234+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbbc810a6e0 con 0x7fbbc8106e20 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.234+0000 7fbbb77fe700 1 -- 172.21.15.98:0/3066743419 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fbbb800b3e0 con 0x7fbbc8106e20 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 msgr2=0x7fbbb003ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 0x7fbbb003ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006fd0 tx=0x7fbbbc006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 msgr2=0x7fbbc819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8006af0 tx=0x7fbbb8011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 shutdown_connections 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fbbb0038660 0x7fbbb003ab20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 --2- 172.21.15.98:0/3066743419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc8106e20 0x7fbbc819a030 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.236+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 >> 172.21.15.98:0/3066743419 conn(0x7fbbc8100870 msgr2=0x7fbbc81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:17.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.237+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 shutdown_connections 2024-03-18T20:19:17.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:15.237+0000 7fbbccc5d700 1 -- 172.21.15.98:0/3066743419 wait complete. 2024-03-18T20:19:17.325 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-03-18T20:19:18.144 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:17 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/864108632' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T20:19:18.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f332750b700 1 Processor -- start 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f332750b700 1 -- start start 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f332750b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f332750b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320079e50 con 0x7f3320106e60 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f33252a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f33252a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57562/0 (socket says 172.21.15.98:57562) 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f33252a7700 1 -- 172.21.15.98:0/2885992700 learned_addr learned my addr 172.21.15.98:0/2885992700 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.629+0000 7f33252a7700 1 -- 172.21.15.98:0/2885992700 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320109790 con 0x7f3320106e60 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f33252a7700 1 --2- 172.21.15.98:0/2885992700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f331c0098a0 tx=0x7f331c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233ea8b1c0703f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f3317fff700 1 -- 172.21.15.98:0/2885992700 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f331c004160 con 0x7f3320106e60 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f3317fff700 1 -- 172.21.15.98:0/2885992700 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f331c0042c0 con 0x7f3320106e60 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f3317fff700 1 -- 172.21.15.98:0/2885992700 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f331c011940 con 0x7f3320106e60 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 -- 172.21.15.98:0/2885992700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 msgr2=0x7f3320109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 --2- 172.21.15.98:0/2885992700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f331c0098a0 tx=0x7f331c009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 -- 172.21.15.98:0/2885992700 shutdown_connections 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 --2- 172.21.15.98:0/2885992700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320109250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 -- 172.21.15.98:0/2885992700 >> 172.21.15.98:0/2885992700 conn(0x7f33201008b0 msgr2=0x7f3320102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 -- 172.21.15.98:0/2885992700 shutdown_connections 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.630+0000 7f332750b700 1 -- 172.21.15.98:0/2885992700 wait complete. 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f332750b700 1 Processor -- start 2024-03-18T20:19:18.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f332750b700 1 -- start start 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f332750b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f332750b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332019a530 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f33252a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f33252a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57572/0 (socket says 172.21.15.98:57572) 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f33252a7700 1 -- 172.21.15.98:0/864108632 learned_addr learned my addr 172.21.15.98:0/864108632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.631+0000 7f33252a7700 1 -- 172.21.15.98:0/864108632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c009550 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f33252a7700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f331c006af0 tx=0x7f331c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f331c003bf0 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f331c01b8b0 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f331c023830 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332019a730 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.632+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332019abd0 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f331c01b440 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332010a4d0 con 0x7f3320106e60 2024-03-18T20:19:18.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f33167fc700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 0x7f330c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f331c04cb60 con 0x7f3320106e60 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f3324aa6700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 0x7f330c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.633+0000 7f3324aa6700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 0x7f330c03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.638+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f331c01b6f0 con 0x7f3320106e60 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.827+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f332010a6c0 con 0x7f3320106e60 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.828+0000 7f33167fc700 1 -- 172.21.15.98:0/864108632 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f331c02b350 con 0x7f3320106e60 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.829+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 msgr2=0x7f330c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.829+0000 7f332750b700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 0x7f330c03ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.829+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 msgr2=0x7f3320199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f331c006af0 tx=0x7f331c011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 shutdown_connections 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f330c0386b0 0x7f330c03ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 --2- 172.21.15.98:0/864108632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3320106e60 0x7f3320199ff0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:18.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 >> 172.21.15.98:0/864108632 conn(0x7f33201008b0 msgr2=0x7f33201013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:18.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 shutdown_connections 2024-03-18T20:19:18.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:17.830+0000 7f332750b700 1 -- 172.21.15.98:0/864108632 wait complete. 2024-03-18T20:19:18.229 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-03-18T20:19:19.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:18 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2523259312' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T20:19:19.161 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: ignoring --setuser ceph since I am not root 2024-03-18T20:19:19.162 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: ignoring --setgroup ceph since I am not root 2024-03-18T20:19:19.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11b24f4700 1 Processor -- start 2024-03-18T20:19:19.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11b24f4700 1 -- start start 2024-03-18T20:19:19.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11b24f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11b24f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac079e10 con 0x7f11ac106eb0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11abfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11abfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57578/0 (socket says 172.21.15.98:57578) 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.552+0000 7f11abfff700 1 -- 172.21.15.98:0/3459826801 learned_addr learned my addr 172.21.15.98:0/3459826801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11abfff700 1 -- 172.21.15.98:0/3459826801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac1097e0 con 0x7f11ac106eb0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11abfff700 1 --2- 172.21.15.98:0/3459826801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f119c0098a0 tx=0x7f119c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d6f56c59ba326a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11aaffd700 1 -- 172.21.15.98:0/3459826801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f119c004160 con 0x7f11ac106eb0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11aaffd700 1 -- 172.21.15.98:0/3459826801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f119c0042c0 con 0x7f11ac106eb0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11aaffd700 1 -- 172.21.15.98:0/3459826801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f119c011940 con 0x7f11ac106eb0 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 -- 172.21.15.98:0/3459826801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 msgr2=0x7f11ac1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 --2- 172.21.15.98:0/3459826801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f119c0098a0 tx=0x7f119c009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 -- 172.21.15.98:0/3459826801 shutdown_connections 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 --2- 172.21.15.98:0/3459826801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1092a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:19.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 -- 172.21.15.98:0/3459826801 >> 172.21.15.98:0/3459826801 conn(0x7f11ac1008a0 msgr2=0x7f11ac102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.553+0000 7f11b24f4700 1 -- 172.21.15.98:0/3459826801 shutdown_connections 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11b24f4700 1 -- 172.21.15.98:0/3459826801 wait complete. 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11b24f4700 1 Processor -- start 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11b24f4700 1 -- start start 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11b24f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11b24f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac106c20 con 0x7f11ac106eb0 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.554+0000 7f11abfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11abfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57592/0 (socket says 172.21.15.98:57592) 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11abfff700 1 -- 172.21.15.98:0/2523259312 learned_addr learned my addr 172.21.15.98:0/2523259312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11abfff700 1 -- 172.21.15.98:0/2523259312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c009550 con 0x7f11ac106eb0 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11abfff700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f119c003f20 tx=0x7f119c011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f119c0037d0 con 0x7f11ac106eb0 2024-03-18T20:19:19.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f119c01b440 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f119c023590 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac104e60 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.555+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac105300 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f119c023800 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11a97fa700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 0x7f119403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f119c04c620 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac10a4d0 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11ab7fe700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 0x7f119403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.556+0000 7f11ab7fe700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 0x7f119403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11a0006fd0 tx=0x7f11a0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.561+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f119c023ab0 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.804+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f11ac10a6c0 con 0x7f11ac106eb0 2024-03-18T20:19:19.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.882+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f119c02c910 con 0x7f11ac106eb0 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.883+0000 7f11a97fa700 1 -- 172.21.15.98:0/2523259312 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f119c04c040 con 0x7f11ac106eb0 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 msgr2=0x7f119403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 0x7f119403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11a0006fd0 tx=0x7f11a0006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 msgr2=0x7f11ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f119c003f20 tx=0x7f119c011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 shutdown_connections 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.886+0000 7f11b24f4700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f1194038700 0x7f119403abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.887+0000 7f11b24f4700 1 --2- 172.21.15.98:0/2523259312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ac106eb0 0x7f11ac1066e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.887+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 >> 172.21.15.98:0/2523259312 conn(0x7f11ac1008a0 msgr2=0x7f11ac101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.887+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 shutdown_connections 2024-03-18T20:19:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:18.887+0000 7f11b24f4700 1 -- 172.21.15.98:0/2523259312 wait complete. 2024-03-18T20:19:19.425 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:19.321+0000 7f573e407200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:19:19.842 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:19.421+0000 7f573e407200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:19:19.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:19.512+0000 7f573e407200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:19:19.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:19.601+0000 7f573e407200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:19:20.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2523259312' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T20:19:20.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-mon[43462]: mgrmap e5: x(active, since 9s) 2024-03-18T20:19:20.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:19 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3476140091' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T20:19:20.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:20.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T20:19:20.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T20:19:20.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.587+0000 7fa5de9e7700 1 Processor -- start 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.587+0000 7fa5de9e7700 1 -- start start 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.587+0000 7fa5de9e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.587+0000 7fa5de9e7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d8079e30 con 0x7fa5d8106e80 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.587+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57624/0 (socket says 172.21.15.98:57624) 2024-03-18T20:19:20.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d7fff700 1 -- 172.21.15.98:0/1815445455 learned_addr learned my addr 172.21.15.98:0/1815445455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d7fff700 1 -- 172.21.15.98:0/1815445455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d81097b0 con 0x7fa5d8106e80 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d7fff700 1 --2- 172.21.15.98:0/1815445455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80098a0 tx=0x7fa5c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d08dedb0e9c3464 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d77fe700 1 -- 172.21.15.98:0/1815445455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c8004160 con 0x7fa5d8106e80 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d77fe700 1 -- 172.21.15.98:0/1815445455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa5c80042c0 con 0x7fa5d8106e80 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.588+0000 7fa5d77fe700 1 -- 172.21.15.98:0/1815445455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c8011940 con 0x7fa5d8106e80 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- 172.21.15.98:0/1815445455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 msgr2=0x7fa5d8109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/1815445455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80098a0 tx=0x7fa5c8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- 172.21.15.98:0/1815445455 shutdown_connections 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/1815445455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d8109270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- 172.21.15.98:0/1815445455 >> 172.21.15.98:0/1815445455 conn(0x7fa5d81008b0 msgr2=0x7fa5d8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- 172.21.15.98:0/1815445455 shutdown_connections 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- 172.21.15.98:0/1815445455 wait complete. 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 Processor -- start 2024-03-18T20:19:20.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.589+0000 7fa5de9e7700 1 -- start start 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5de9e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5de9e7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d8106bf0 con 0x7fa5d8106e80 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57628/0 (socket says 172.21.15.98:57628) 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d7fff700 1 -- 172.21.15.98:0/3476140091 learned_addr learned my addr 172.21.15.98:0/3476140091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d7fff700 1 -- 172.21.15.98:0/3476140091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8009550 con 0x7fa5d8106e80 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d7fff700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8003f20 tx=0x7fa5c8011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c80037d0 con 0x7fa5d8106e80 2024-03-18T20:19:20.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa5c801b440 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d8104e30 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5c8023590 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.590+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d81052d0 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.591+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d810a4f0 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.592+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fa5c801b5b0 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.592+0000 7fa5d5ffb700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 0x7fa5c403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.592+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa5c804cd00 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.592+0000 7fa5cffff700 1 -- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 msgr2=0x7fa5c403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.592+0000 7fa5cffff700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 0x7fa5c403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.596+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa5c801b920 con 0x7fa5d8106e80 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.793+0000 7fa5cffff700 1 -- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 msgr2=0x7fa5c403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:20.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.793+0000 7fa5cffff700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 0x7fa5c403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.850+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa5d8105930 con 0x7fa5d8106e80 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.850+0000 7fa5d5ffb700 1 -- 172.21.15.98:0/3476140091 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa5c804a070 con 0x7fa5d8106e80 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 msgr2=0x7fa5c403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 0x7fa5c403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 msgr2=0x7fa5d81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8003f20 tx=0x7fa5c8011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 shutdown_connections 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7fa5c4038750 0x7fa5c403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 --2- 172.21.15.98:0/3476140091 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5d8106e80 0x7fa5d81066b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 >> 172.21.15.98:0/3476140091 conn(0x7fa5d81008b0 msgr2=0x7fa5d8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 shutdown_connections 2024-03-18T20:19:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:19.853+0000 7fa5de9e7700 1 -- 172.21.15.98:0/3476140091 wait complete. 2024-03-18T20:19:20.212 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-03-18T20:19:20.212 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-03-18T20:19:20.533 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:20.251+0000 7f573e407200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:20.842 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:20.529+0000 7f573e407200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:19:20.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:20.617+0000 7f573e407200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:19:20.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:20.725+0000 7f573e407200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:21.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:20.882+0000 7f573e407200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:19:21.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:21.006+0000 7f573e407200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:19:21.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:21.090+0000 7f573e407200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:19:21.839 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:21.455+0000 7f573e407200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:19:22.092 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:21.836+0000 7f573e407200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:19:22.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:22.027+0000 7f573e407200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:19:22.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:22.118+0000 7f573e407200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:19:23.342 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:22.873+0000 7f573e407200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:19:23.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:22.970+0000 7f573e407200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:19:23.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:23.116+0000 7f573e407200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:19:24.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:23.734+0000 7f573e407200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:19:24.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:23.916+0000 7f573e407200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:19:24.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:24 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:24.108+0000 7f573e407200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:19:25.300 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:25.017+0000 7f573e407200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:19:25.575 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:25.296+0000 7f573e407200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:25.576 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:25.383+0000 7f573e407200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:25.571+0000 7f573e407200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: Active manager daemon x restarted 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: Activating manager daemon x 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: mgrmap e6: x(active, starting, since 0.00321997s) 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:19:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: Manager daemon x is now available 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:25.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:25 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:26.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:26.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.545+0000 7f2a78848700 1 Processor -- start 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.545+0000 7f2a78848700 1 -- start start 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.545+0000 7f2a78848700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.545+0000 7f2a78848700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a74106db0 con 0x7f2a741063c0 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.545+0000 7f2a7259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57638/0 (socket says 172.21.15.98:57638) 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7259c700 1 -- 172.21.15.98:0/2912321527 learned_addr learned my addr 172.21.15.98:0/2912321527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7259c700 1 -- 172.21.15.98:0/2912321527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74107570 con 0x7f2a741063c0 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7259c700 1 --2- 172.21.15.98:0/2912321527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098a0 tx=0x7f2a64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255534cd64a7da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:26.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7159a700 1 -- 172.21.15.98:0/2912321527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a64004160 con 0x7f2a741063c0 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7159a700 1 -- 172.21.15.98:0/2912321527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2a640042c0 con 0x7f2a741063c0 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.546+0000 7f2a7159a700 1 -- 172.21.15.98:0/2912321527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a64011940 con 0x7f2a741063c0 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 -- 172.21.15.98:0/2912321527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 msgr2=0x7f2a741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 --2- 172.21.15.98:0/2912321527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098a0 tx=0x7f2a64009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 -- 172.21.15.98:0/2912321527 shutdown_connections 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 --2- 172.21.15.98:0/2912321527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a741067e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 -- 172.21.15.98:0/2912321527 >> 172.21.15.98:0/2912321527 conn(0x7f2a741019a0 msgr2=0x7f2a74103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 -- 172.21.15.98:0/2912321527 shutdown_connections 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.547+0000 7f2a78848700 1 -- 172.21.15.98:0/2912321527 wait complete. 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a78848700 1 Processor -- start 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a78848700 1 -- start start 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a78848700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:26.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a78848700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7419a5d0 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a7259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a7259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57646/0 (socket says 172.21.15.98:57646) 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.548+0000 7f2a7259c700 1 -- 172.21.15.98:0/3076716183 learned_addr learned my addr 172.21.15.98:0/3076716183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a7259c700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64009550 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a7259c700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a64003f20 tx=0x7f2a64011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a640036a0 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2a64003c70 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a6401be80 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7419a7d0 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.549+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7419ac70 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.550+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f2a6401b440 con 0x7f2a741063c0 2024-03-18T20:19:26.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.551+0000 7f2a637fe700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.551+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a5c03b320 con 0x7f2a5c038750 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.551+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2a6404c5d0 con 0x7f2a741063c0 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.551+0000 7f2a71d9b700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.551+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.752+0000 7f2a71d9b700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:20.752+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:21.153+0000 7f2a71d9b700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:21.153+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:21.954+0000 7f2a71d9b700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:21.954+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:23.555+0000 7f2a71d9b700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/479479747 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:23.555+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:25.587+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) v1 ==== 45300+0+0 (secure 0 0 0) 0x7f2a640234e0 con 0x7f2a741063c0 2024-03-18T20:19:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:25.587+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 msgr2=0x7f2a5c03ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:25.587+0000 7f2a637fe700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.588+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f2a6404dff0 con 0x7f2a741063c0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.588+0000 7f2a637fe700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 0x7f2a5c03e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.588+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a5c03b320 con 0x7f2a5c03bfe0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.588+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 0x7f2a5c03e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.589+0000 7f2a71d9b700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 0x7f2a5c03e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a68003a10 tx=0x7f2a680092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.589+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f2a5c03b320 con 0x7f2a5c03bfe0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.595+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2a7419ae00 con 0x7f2a5c03bfe0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a637fe700 1 -- 172.21.15.98:0/3076716183 <== mgr.14120 v2:172.21.15.98:6800/1287611061 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2a6800a050 con 0x7f2a5c03bfe0 2024-03-18T20:19:26.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 msgr2=0x7f2a5c03e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 0x7f2a5c03e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a68003a10 tx=0x7f2a680092b0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 msgr2=0x7f2a7419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a64003f20 tx=0x7f2a64011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 shutdown_connections 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f2a5c03bfe0 0x7f2a5c03e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a741063c0 0x7f2a7419a090 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 --2- 172.21.15.98:0/3076716183 >> [v2:172.21.15.98:6800/479479747,v1:172.21.15.98:6801/479479747] conn(0x7f2a5c038750 0x7f2a5c03ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 >> 172.21.15.98:0/3076716183 conn(0x7f2a741019a0 msgr2=0x7f2a74102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 shutdown_connections 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:26.596+0000 7f2a78848700 1 -- 172.21.15.98:0/3076716183 wait complete. 2024-03-18T20:19:26.971 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-03-18T20:19:26.972 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-03-18T20:19:27.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: Found migration_current of "None". Setting to last migration. 2024-03-18T20:19:27.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:19:27.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:19:27.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:27.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:27.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:26 smithi098 ceph-mon[43462]: mgrmap e7: x(active, since 1.00508s) 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:26] ENGINE Bus STARTING 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:27 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:28.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.292+0000 7ff9a0b09700 1 Processor -- start 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.292+0000 7ff9a0b09700 1 -- start start 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff9a0b09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff9a0b09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c079e50 con 0x7ff99c106e60 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34354/0 (socket says 172.21.15.98:34354) 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff99a59c700 1 -- 172.21.15.98:0/462357949 learned_addr learned my addr 172.21.15.98:0/462357949 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:28.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff99a59c700 1 -- 172.21.15.98:0/462357949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c109790 con 0x7ff99c106e60 2024-03-18T20:19:28.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.293+0000 7ff99a59c700 1 --2- 172.21.15.98:0/462357949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0098a0 tx=0x7ff98c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d4c193d0900c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff99959a700 1 -- 172.21.15.98:0/462357949 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff98c004160 con 0x7ff99c106e60 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff99959a700 1 -- 172.21.15.98:0/462357949 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff98c0042c0 con 0x7ff99c106e60 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff99959a700 1 -- 172.21.15.98:0/462357949 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff98c011940 con 0x7ff99c106e60 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 -- 172.21.15.98:0/462357949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 msgr2=0x7ff99c109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/462357949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0098a0 tx=0x7ff98c009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 -- 172.21.15.98:0/462357949 shutdown_connections 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/462357949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c109250 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 -- 172.21.15.98:0/462357949 >> 172.21.15.98:0/462357949 conn(0x7ff99c1008f0 msgr2=0x7ff99c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 -- 172.21.15.98:0/462357949 shutdown_connections 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.294+0000 7ff9a0b09700 1 -- 172.21.15.98:0/462357949 wait complete. 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff9a0b09700 1 Processor -- start 2024-03-18T20:19:28.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff9a0b09700 1 -- start start 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff9a0b09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff9a0b09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c19a580 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34370/0 (socket says 172.21.15.98:34370) 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.295+0000 7ff99a59c700 1 -- 172.21.15.98:0/2722954341 learned_addr learned my addr 172.21.15.98:0/2722954341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff99a59c700 1 -- 172.21.15.98:0/2722954341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c009550 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff99a59c700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff98c006af0 tx=0x7ff98c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff98c003bf0 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff98c01b8b0 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c19a780 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff98c023800 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.296+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c19ac20 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7ff98c023960 con 0x7ff99c106e60 2024-03-18T20:19:28.099 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c10a4d0 con 0x7ff99c106e60 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff98b7fe700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 0x7ff98403aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff98c04ca00 con 0x7ff99c106e60 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff999d9b700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 0x7ff98403aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.297+0000 7ff999d9b700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 0x7ff98403aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff990006fd0 tx=0x7ff990006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.302+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff98c023c10 con 0x7ff99c106e60 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.508+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff99c101af0 con 0x7ff9840385e0 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.515+0000 7ff98b7fe700 1 -- 172.21.15.98:0/2722954341 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff99c101af0 con 0x7ff9840385e0 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 msgr2=0x7ff98403aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 0x7ff98403aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff990006fd0 tx=0x7ff990006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 msgr2=0x7ff99c19a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff98c006af0 tx=0x7ff98c011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 shutdown_connections 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7ff9840385e0 0x7ff98403aaa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.100 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 --2- 172.21.15.98:0/2722954341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c106e60 0x7ff99c19a040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.101 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 >> 172.21.15.98:0/2722954341 conn(0x7ff99c1008f0 msgr2=0x7ff99c1013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:28.101 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 shutdown_connections 2024-03-18T20:19:28.101 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:27.517+0000 7ff9a0b09700 1 -- 172.21.15.98:0/2722954341 wait complete. 2024-03-18T20:19:28.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T20:19:28.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.408+0000 7fd2599cc700 1 Processor -- start 2024-03-18T20:19:28.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.408+0000 7fd2599cc700 1 -- start start 2024-03-18T20:19:28.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.408+0000 7fd2599cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd2599cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd254079f50 con 0x7fd254106ee0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd252ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd252ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34384/0 (socket says 172.21.15.98:34384) 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd252ffd700 1 -- 172.21.15.98:0/822924746 learned_addr learned my addr 172.21.15.98:0/822924746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd252ffd700 1 -- 172.21.15.98:0/822924746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd254109810 con 0x7fd254106ee0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd252ffd700 1 --2- 172.21.15.98:0/822924746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd2480098a0 tx=0x7fd248009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ede2d14fd8a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd251ffb700 1 -- 172.21.15.98:0/822924746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd248004160 con 0x7fd254106ee0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd251ffb700 1 -- 172.21.15.98:0/822924746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd2480042c0 con 0x7fd254106ee0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.409+0000 7fd251ffb700 1 -- 172.21.15.98:0/822924746 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd248011940 con 0x7fd254106ee0 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 -- 172.21.15.98:0/822924746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 msgr2=0x7fd2541092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 --2- 172.21.15.98:0/822924746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd2480098a0 tx=0x7fd248009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 -- 172.21.15.98:0/822924746 shutdown_connections 2024-03-18T20:19:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 --2- 172.21.15.98:0/822924746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd2541092d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 -- 172.21.15.98:0/822924746 >> 172.21.15.98:0/822924746 conn(0x7fd254100950 msgr2=0x7fd254102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 -- 172.21.15.98:0/822924746 shutdown_connections 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.410+0000 7fd2599cc700 1 -- 172.21.15.98:0/822924746 wait complete. 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd2599cc700 1 Processor -- start 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd2599cc700 1 -- start start 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd2599cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd2599cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25419a650 con 0x7fd254106ee0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd252ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd252ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34386/0 (socket says 172.21.15.98:34386) 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd252ffd700 1 -- 172.21.15.98:0/617493171 learned_addr learned my addr 172.21.15.98:0/617493171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd252ffd700 1 -- 172.21.15.98:0/617493171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd248009550 con 0x7fd254106ee0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.411+0000 7fd252ffd700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd248006af0 tx=0x7fd248011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.412+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd248003bf0 con 0x7fd254106ee0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.412+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd24801b8b0 con 0x7fd254106ee0 2024-03-18T20:19:28.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.412+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25419a850 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.412+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd248023800 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.412+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25419acf0 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7fd248023a70 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2589ca700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 0x7fd23c03aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25410a600 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd24804cb50 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2527fc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 0x7fd23c03aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.413+0000 7fd2527fc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 0x7fd23c03aa50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd244006fd0 tx=0x7fd244006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.418+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd248023d20 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.513+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fd24802cc90 con 0x7fd254106ee0 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.612+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd254101b30 con 0x7fd23c038590 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.613+0000 7fd2589ca700 1 -- 172.21.15.98:0/617493171 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd254101b30 con 0x7fd23c038590 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 msgr2=0x7fd23c03aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 0x7fd23c03aa50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd244006fd0 tx=0x7fd244006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 msgr2=0x7fd25419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd248006af0 tx=0x7fd248011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 shutdown_connections 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fd23c038590 0x7fd23c03aa50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 --2- 172.21.15.98:0/617493171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd254106ee0 0x7fd25419a110 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 >> 172.21.15.98:0/617493171 conn(0x7fd254100950 msgr2=0x7fd254101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 shutdown_connections 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:28.615+0000 7fd2599cc700 1 -- 172.21.15.98:0/617493171 wait complete. 2024-03-18T20:19:28.991 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-03-18T20:19:29.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:28 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:26] ENGINE Serving on http://172.21.15.98:8765 2024-03-18T20:19:29.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:28 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:27] ENGINE Serving on https://172.21.15.98:7150 2024-03-18T20:19:29.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:28 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:27] ENGINE Bus STARTED 2024-03-18T20:19:29.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:28 smithi098 ceph-mon[43462]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:29.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:28 smithi098 ceph-mon[43462]: mgrmap e8: x(active, since 2s) 2024-03-18T20:19:29.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-mon[43462]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: Generating public/private rsa key pair. 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: Your identification has been saved in /tmp/tmpaxamrm5h/key. 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: Your public key has been saved in /tmp/tmpaxamrm5h/key.pub. 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: The key fingerprint is: 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: SHA256:dNdXdV1sKzqXsgtHLT++1uOcKd2dygP64yCPQ0qoh+A ceph-a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: The key's randomart image is: 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: +---[RSA 3072]----+ 2024-03-18T20:19:29.910 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | .O| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | . *| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | . . . ..o| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | . . ... o | 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | . S o..o | 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: |. . . . .=oo | 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: |o o . o. o..*oo +| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | E . . .+.+oo+o==| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: | . ...o++=*=.| 2024-03-18T20:19:29.911 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:29 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: +----[SHA256]-----+ 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed7b25700 1 Processor -- start 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed7b25700 1 -- start start 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed7b25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed7b25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed0106db0 con 0x7efed01063c0 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed58c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed58c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34390/0 (socket says 172.21.15.98:34390) 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed58c1700 1 -- 172.21.15.98:0/1002518730 learned_addr learned my addr 172.21.15.98:0/1002518730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:29.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.307+0000 7efed58c1700 1 -- 172.21.15.98:0/1002518730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0107570 con 0x7efed01063c0 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed58c1700 1 --2- 172.21.15.98:0/1002518730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efecc0098a0 tx=0x7efecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5227d58621c8a59a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed48bf700 1 -- 172.21.15.98:0/1002518730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efecc004160 con 0x7efed01063c0 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed48bf700 1 -- 172.21.15.98:0/1002518730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efecc0042c0 con 0x7efed01063c0 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed48bf700 1 -- 172.21.15.98:0/1002518730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efecc011940 con 0x7efed01063c0 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 -- 172.21.15.98:0/1002518730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 msgr2=0x7efed01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 --2- 172.21.15.98:0/1002518730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efecc0098a0 tx=0x7efecc009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 -- 172.21.15.98:0/1002518730 shutdown_connections 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 --2- 172.21.15.98:0/1002518730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed01067e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 -- 172.21.15.98:0/1002518730 >> 172.21.15.98:0/1002518730 conn(0x7efed01019a0 msgr2=0x7efed0103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 -- 172.21.15.98:0/1002518730 shutdown_connections 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.308+0000 7efed7b25700 1 -- 172.21.15.98:0/1002518730 wait complete. 2024-03-18T20:19:29.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed7b25700 1 Processor -- start 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed7b25700 1 -- start start 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed7b25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed7b25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed019a5d0 con 0x7efed01063c0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed58c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed58c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34396/0 (socket says 172.21.15.98:34396) 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed58c1700 1 -- 172.21.15.98:0/976189358 learned_addr learned my addr 172.21.15.98:0/976189358 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.309+0000 7efed58c1700 1 -- 172.21.15.98:0/976189358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc009550 con 0x7efed01063c0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efed58c1700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efecc003f20 tx=0x7efecc011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efecc0036a0 con 0x7efed01063c0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efecc003c70 con 0x7efed01063c0 2024-03-18T20:19:29.998 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed019a7d0 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efecc01be80 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.310+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed019ac70 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7efecc0234e0 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efec6ffd700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 0x7efebc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed010a7a0 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efecc04c860 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efed50c0700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 0x7efebc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.311+0000 7efed50c0700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 0x7efebc03abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.316+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efecc023cb0 con 0x7efed01063c0 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.507+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efed0102c10 con 0x7efebc038700 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.629+0000 7efec6ffd700 1 -- 172.21.15.98:0/976189358 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efed0102c10 con 0x7efebc038700 2024-03-18T20:19:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 msgr2=0x7efebc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 0x7efebc03abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 msgr2=0x7efed019a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efecc003f20 tx=0x7efecc011f00 comp rx=0 tx=0).stop 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 shutdown_connections 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efebc038700 0x7efebc03abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 --2- 172.21.15.98:0/976189358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed01063c0 0x7efed019a090 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 >> 172.21.15.98:0/976189358 conn(0x7efed01019a0 msgr2=0x7efed0102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 shutdown_connections 2024-03-18T20:19:30.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:29.631+0000 7efed7b25700 1 -- 172.21.15.98:0/976189358 wait complete. 2024-03-18T20:19:30.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:30 smithi098 ceph-mon[43462]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:30.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:30 smithi098 ceph-mon[43462]: Generating ssh key... 2024-03-18T20:19:30.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:30.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.315+0000 7efe70a79700 1 Processor -- start 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.315+0000 7efe70a79700 1 -- start start 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe70a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe70a79700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c079e10 con 0x7efe6c104c90 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34400/0 (socket says 172.21.15.98:34400) 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6a59c700 1 -- 172.21.15.98:0/1039248916 learned_addr learned my addr 172.21.15.98:0/1039248916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6a59c700 1 -- 172.21.15.98:0/1039248916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c1055f0 con 0x7efe6c104c90 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6a59c700 1 --2- 172.21.15.98:0/1039248916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe600098a0 tx=0x7efe60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b456421bc4c63644 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6959a700 1 -- 172.21.15.98:0/1039248916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe60004160 con 0x7efe6c104c90 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6959a700 1 -- 172.21.15.98:0/1039248916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efe600042c0 con 0x7efe6c104c90 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.316+0000 7efe6959a700 1 -- 172.21.15.98:0/1039248916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe60011940 con 0x7efe6c104c90 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 -- 172.21.15.98:0/1039248916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 msgr2=0x7efe6c1050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:30.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 --2- 172.21.15.98:0/1039248916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe600098a0 tx=0x7efe60009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 -- 172.21.15.98:0/1039248916 shutdown_connections 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 --2- 172.21.15.98:0/1039248916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c1050b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 -- 172.21.15.98:0/1039248916 >> 172.21.15.98:0/1039248916 conn(0x7efe6c1008b0 msgr2=0x7efe6c102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 -- 172.21.15.98:0/1039248916 shutdown_connections 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.317+0000 7efe70a79700 1 -- 172.21.15.98:0/1039248916 wait complete. 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe70a79700 1 Processor -- start 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe70a79700 1 -- start start 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe70a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe70a79700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c19c720 con 0x7efe6c104c90 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34414/0 (socket says 172.21.15.98:34414) 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe6a59c700 1 -- 172.21.15.98:0/919432829 learned_addr learned my addr 172.21.15.98:0/919432829 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.318+0000 7efe6a59c700 1 -- 172.21.15.98:0/919432829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60009550 con 0x7efe6c104c90 2024-03-18T20:19:30.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe6a59c700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe60006af0 tx=0x7efe60011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe60003bf0 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c19c920 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efe6001b8b0 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe60023830 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c19cdc0 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.319+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7efe60023af0 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.320+0000 7efe5b7fe700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 0x7efe5403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.320+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe6004cc30 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.320+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c195f10 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.320+0000 7efe69d9b700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 0x7efe5403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.320+0000 7efe69d9b700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 0x7efe5403ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe5c006fd0 tx=0x7efe5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.325+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efe60023da0 con 0x7efe6c104c90 2024-03-18T20:19:30.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.514+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efe6c0611f0 con 0x7efe540386b0 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.515+0000 7efe5b7fe700 1 -- 172.21.15.98:0/919432829 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7efe6c0611f0 con 0x7efe540386b0 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 msgr2=0x7efe5403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 0x7efe5403ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe5c006fd0 tx=0x7efe5c006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 msgr2=0x7efe6c19c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe60006af0 tx=0x7efe60011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 shutdown_connections 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7efe540386b0 0x7efe5403ab70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 --2- 172.21.15.98:0/919432829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe6c104c90 0x7efe6c19c1e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 >> 172.21.15.98:0/919432829 conn(0x7efe6c1008b0 msgr2=0x7efe6c193050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 shutdown_connections 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:30.517+0000 7efe70a79700 1 -- 172.21.15.98:0/919432829 wait complete. 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T20:19:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-03-18T20:19:31.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:31 smithi098 ceph-mon[43462]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:33.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:32 smithi098 ceph-mon[43462]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:33.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:32 smithi098 ceph-mon[43462]: Deploying cephadm binary to smithi098 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdc17bd700 1 Processor -- start 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdc17bd700 1 -- start start 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdc17bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdc17bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc079e50 con 0x7fcdbc106de0 2024-03-18T20:19:33.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34418/0 (socket says 172.21.15.98:34418) 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.185+0000 7fcdbaffd700 1 -- 172.21.15.98:0/1536327528 learned_addr learned my addr 172.21.15.98:0/1536327528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.186+0000 7fcdbaffd700 1 -- 172.21.15.98:0/1536327528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc109710 con 0x7fcdbc106de0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.186+0000 7fcdbaffd700 1 --2- 172.21.15.98:0/1536327528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00098a0 tx=0x7fcdb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c208bb628e7d94b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.186+0000 7fcdb9ffb700 1 -- 172.21.15.98:0/1536327528 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0004160 con 0x7fcdbc106de0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.186+0000 7fcdb9ffb700 1 -- 172.21.15.98:0/1536327528 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fcdb00042c0 con 0x7fcdbc106de0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.186+0000 7fcdb9ffb700 1 -- 172.21.15.98:0/1536327528 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0011940 con 0x7fcdbc106de0 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- 172.21.15.98:0/1536327528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 msgr2=0x7fcdbc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/1536327528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00098a0 tx=0x7fcdb0009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- 172.21.15.98:0/1536327528 shutdown_connections 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/1536327528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc1091d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- 172.21.15.98:0/1536327528 >> 172.21.15.98:0/1536327528 conn(0x7fcdbc100810 msgr2=0x7fcdbc102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- 172.21.15.98:0/1536327528 shutdown_connections 2024-03-18T20:19:33.808 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- 172.21.15.98:0/1536327528 wait complete. 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 Processor -- start 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.187+0000 7fcdc17bd700 1 -- start start 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdc17bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdc17bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc19a510 con 0x7fcdbc106de0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34432/0 (socket says 172.21.15.98:34432) 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdbaffd700 1 -- 172.21.15.98:0/546589019 learned_addr learned my addr 172.21.15.98:0/546589019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdbaffd700 1 -- 172.21.15.98:0/546589019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb0009550 con 0x7fcdbc106de0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdbaffd700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0006af0 tx=0x7fcdb0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0003bf0 con 0x7fcdbc106de0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc19a710 con 0x7fcdbc106de0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fcdb001b8b0 con 0x7fcdbc106de0 2024-03-18T20:19:33.809 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0023800 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.188+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc19abb0 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.189+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fcdb0023a70 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.189+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc10a450 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.189+0000 7fcda3fff700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 0x7fcda403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.189+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcdb004cc70 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.189+0000 7fcdba7fc700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 0x7fcda403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.190+0000 7fcdba7fc700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 0x7fcda403ab70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006fd0 tx=0x7fcdac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.194+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcdb0018760 con 0x7fcdbc106de0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:31.387+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdbc101ad0 con 0x7fcda40386b0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.501+0000 7fcda3fff700 1 -- 172.21.15.98:0/546589019 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fcdbc101ad0 con 0x7fcda40386b0 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 msgr2=0x7fcda403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 0x7fcda403ab70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006fd0 tx=0x7fcdac006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 msgr2=0x7fcdbc199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0006af0 tx=0x7fcdb0011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 shutdown_connections 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fcda40386b0 0x7fcda403ab70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 --2- 172.21.15.98:0/546589019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdbc106de0 0x7fcdbc199fd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 >> 172.21.15.98:0/546589019 conn(0x7fcdbc100810 msgr2=0x7fcdbc1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 shutdown_connections 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:33.504+0000 7fcdc17bd700 1 -- 172.21.15.98:0/546589019 wait complete. 2024-03-18T20:19:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-03-18T20:19:34.779 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T20:19:34.779 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.140+0000 7f4a6c782700 1 Processor -- start 2024-03-18T20:19:34.779 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.140+0000 7f4a6c782700 1 -- start start 2024-03-18T20:19:34.779 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.140+0000 7f4a6c782700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.140+0000 7f4a6c782700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64105060 con 0x7f4a64104670 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6a51e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6a51e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47208/0 (socket says 172.21.15.98:47208) 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6a51e700 1 -- 172.21.15.98:0/2076882527 learned_addr learned my addr 172.21.15.98:0/2076882527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6a51e700 1 -- 172.21.15.98:0/2076882527 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a641051a0 con 0x7f4a64104670 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6a51e700 1 --2- 172.21.15.98:0/2076882527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a600098a0 tx=0x7f4a60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d8c94dffafe244 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6951c700 1 -- 172.21.15.98:0/2076882527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a60004160 con 0x7f4a64104670 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6951c700 1 -- 172.21.15.98:0/2076882527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4a600042c0 con 0x7f4a64104670 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.141+0000 7f4a6951c700 1 -- 172.21.15.98:0/2076882527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a60011940 con 0x7f4a64104670 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 -- 172.21.15.98:0/2076882527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 msgr2=0x7f4a64104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 --2- 172.21.15.98:0/2076882527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a600098a0 tx=0x7f4a60009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 -- 172.21.15.98:0/2076882527 shutdown_connections 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 --2- 172.21.15.98:0/2076882527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a64104a90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 -- 172.21.15.98:0/2076882527 >> 172.21.15.98:0/2076882527 conn(0x7f4a64100090 msgr2=0x7f4a641024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 -- 172.21.15.98:0/2076882527 shutdown_connections 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.142+0000 7f4a6c782700 1 -- 172.21.15.98:0/2076882527 wait complete. 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6c782700 1 Processor -- start 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6c782700 1 -- start start 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6c782700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6c782700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6419a5d0 con 0x7f4a64104670 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6a51e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6a51e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47214/0 (socket says 172.21.15.98:47214) 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6a51e700 1 -- 172.21.15.98:0/947980600 learned_addr learned my addr 172.21.15.98:0/947980600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6a51e700 1 -- 172.21.15.98:0/947980600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60009550 con 0x7f4a64104670 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6a51e700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a60006af0 tx=0x7f4a60011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a60003bf0 con 0x7f4a64104670 2024-03-18T20:19:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4a6001b8b0 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6419a7d0 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.143+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a60023830 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.144+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6419ac70 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.144+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f4a60023aa0 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.144+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a641082d0 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.145+0000 7f4a5b7fe700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 0x7f4a5003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.145+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4a6004c9b0 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.145+0000 7f4a69d1d700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 0x7f4a5003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.145+0000 7f4a69d1d700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 0x7f4a5003ab70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006fd0 tx=0x7f4a54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.149+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a60018350 con 0x7f4a64104670 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.363+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4a641051a0 con 0x7f4a500386b0 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.367+0000 7f4a5b7fe700 1 -- 172.21.15.98:0/947980600 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4a641051a0 con 0x7f4a500386b0 2024-03-18T20:19:34.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 msgr2=0x7f4a5003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 0x7f4a5003ab70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a54006fd0 tx=0x7f4a54006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 msgr2=0x7f4a6419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a60006af0 tx=0x7f4a60011a50 comp rx=0 tx=0).stop 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 shutdown_connections 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.369+0000 7f4a6c782700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4a500386b0 0x7f4a5003ab70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.370+0000 7f4a6c782700 1 --2- 172.21.15.98:0/947980600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a64104670 0x7f4a6419a090 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.370+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 >> 172.21.15.98:0/947980600 conn(0x7f4a64100090 msgr2=0x7f4a64100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.370+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 shutdown_connections 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:34.370+0000 7f4a6c782700 1 -- 172.21.15.98:0/947980600 wait complete. 2024-03-18T20:19:34.783 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-03-18T20:19:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:34 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:34 smithi098 ceph-mon[43462]: Added host smithi098 2024-03-18T20:19:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:34 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:34 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:35.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b4520700 1 Processor -- start 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b4520700 1 -- start start 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b4520700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b4520700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac105cf0 con 0x7f96ac105300 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b22bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b22bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47226/0 (socket says 172.21.15.98:47226) 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.123+0000 7f96b22bc700 1 -- 172.21.15.98:0/1024454216 learned_addr learned my addr 172.21.15.98:0/1024454216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b22bc700 1 -- 172.21.15.98:0/1024454216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac106500 con 0x7f96ac105300 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b22bc700 1 --2- 172.21.15.98:0/1024454216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96a80098a0 tx=0x7f96a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677981b11681117 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b12ba700 1 -- 172.21.15.98:0/1024454216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96a8004160 con 0x7f96ac105300 2024-03-18T20:19:35.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b12ba700 1 -- 172.21.15.98:0/1024454216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f96a80042c0 con 0x7f96ac105300 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b12ba700 1 -- 172.21.15.98:0/1024454216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96a80118e0 con 0x7f96ac105300 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b4520700 1 -- 172.21.15.98:0/1024454216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 msgr2=0x7f96ac105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.124+0000 7f96b4520700 1 --2- 172.21.15.98:0/1024454216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96a80098a0 tx=0x7f96a8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- 172.21.15.98:0/1024454216 shutdown_connections 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 --2- 172.21.15.98:0/1024454216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac105720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- 172.21.15.98:0/1024454216 >> 172.21.15.98:0/1024454216 conn(0x7f96ac100920 msgr2=0x7f96ac102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- 172.21.15.98:0/1024454216 shutdown_connections 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- 172.21.15.98:0/1024454216 wait complete. 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 Processor -- start 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- start start 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.125+0000 7f96b4520700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac079f20 con 0x7f96ac105300 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b22bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:35.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b22bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47238/0 (socket says 172.21.15.98:47238) 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b22bc700 1 -- 172.21.15.98:0/2106257237 learned_addr learned my addr 172.21.15.98:0/2106257237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b22bc700 1 -- 172.21.15.98:0/2106257237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009550 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b22bc700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96a8003f70 tx=0x7f96a80119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96a80037d0 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f96a801b440 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96a80193d0 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac19ce70 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.126+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac19d2b0 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.127+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f96a8003930 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.127+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac0623e0 con 0x7f96ac105300 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.127+0000 7f96a37fe700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 0x7f969803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.128+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f96a804c500 con 0x7f96ac105300 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.128+0000 7f96b1abb700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 0x7f969803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.128+0000 7f96b1abb700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 0x7f969803ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f969c006fd0 tx=0x7f969c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.132+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f96a80189a0 con 0x7f96ac105300 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.366+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f96ac101c30 con 0x7f9698038750 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.370+0000 7f96a37fe700 1 -- 172.21.15.98:0/2106257237 <== mgr.14120 v2:172.21.15.98:6800/1287611061 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f96ac101c30 con 0x7f9698038750 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.372+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 msgr2=0x7f969803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.372+0000 7f96b4520700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 0x7f969803ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f969c006fd0 tx=0x7f969c006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.372+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 msgr2=0x7f96ac0799e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.372+0000 7f96b4520700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96a8003f70 tx=0x7f96a80119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:35.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 shutdown_connections 2024-03-18T20:19:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f9698038750 0x7f969803ac10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 --2- 172.21.15.98:0/2106257237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96ac105300 0x7f96ac0799e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 >> 172.21.15.98:0/2106257237 conn(0x7f96ac100920 msgr2=0x7f96ac101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 shutdown_connections 2024-03-18T20:19:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:35.373+0000 7f96b4520700 1 -- 172.21.15.98:0/2106257237 wait complete. 2024-03-18T20:19:35.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:35 smithi098 ceph-mon[43462]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:35.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:35 smithi098 ceph-mon[43462]: Saving service mon spec with placement count:5 2024-03-18T20:19:35.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:35 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:35.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:35 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:35.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:35 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.192+0000 7fca0e5a4700 1 Processor -- start 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.192+0000 7fca0e5a4700 1 -- start start 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.192+0000 7fca0e5a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.192+0000 7fca0e5a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca08105080 con 0x7fca08104690 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47240/0 (socket says 172.21.15.98:47240) 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca07fff700 1 -- 172.21.15.98:0/3907020904 learned_addr learned my addr 172.21.15.98:0/3907020904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:36.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca07fff700 1 -- 172.21.15.98:0/3907020904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08107620 con 0x7fca08104690 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca07fff700 1 --2- 172.21.15.98:0/3907020904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80098a0 tx=0x7fc9f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7754938645f23b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca06ffd700 1 -- 172.21.15.98:0/3907020904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8004160 con 0x7fca08104690 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca06ffd700 1 -- 172.21.15.98:0/3907020904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9f80042c0 con 0x7fca08104690 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.193+0000 7fca06ffd700 1 -- 172.21.15.98:0/3907020904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f80118e0 con 0x7fca08104690 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3907020904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 msgr2=0x7fca08104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3907020904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80098a0 tx=0x7fc9f8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3907020904 shutdown_connections 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3907020904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca08104ab0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3907020904 >> 172.21.15.98:0/3907020904 conn(0x7fca08100060 msgr2=0x7fca08102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3907020904 shutdown_connections 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.194+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3907020904 wait complete. 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca0e5a4700 1 Processor -- start 2024-03-18T20:19:36.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca0e5a4700 1 -- start start 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca0e5a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca0e5a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0819a620 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47246/0 (socket says 172.21.15.98:47246) 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca07fff700 1 -- 172.21.15.98:0/3603459192 learned_addr learned my addr 172.21.15.98:0/3603459192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.195+0000 7fca07fff700 1 -- 172.21.15.98:0/3603459192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f8009550 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca07fff700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8003f70 tx=0x7fc9f80119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f80036a0 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9f801c760 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0819a820 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8025570 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.196+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0819acc0 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.197+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fc9f80257a0 con 0x7fca08104690 2024-03-18T20:19:36.772 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.197+0000 7fca057fa700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 0x7fc9f003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.197+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc9f804fff0 con 0x7fca08104690 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.197+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0810a850 con 0x7fca08104690 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.197+0000 7fca077fe700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 0x7fc9f003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.198+0000 7fca077fe700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 0x7fc9f003ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.202+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc9f8025aa0 con 0x7fca08104690 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.405+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fca08105830 con 0x7fca08104690 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.409+0000 7fca057fa700 1 -- 172.21.15.98:0/3603459192 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc9f804d020 con 0x7fca08104690 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 msgr2=0x7fc9f003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 0x7fc9f003ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 msgr2=0x7fca0819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8003f70 tx=0x7fc9f80119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 shutdown_connections 2024-03-18T20:19:36.773 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fc9f0038750 0x7fc9f003ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:36.774 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 --2- 172.21.15.98:0/3603459192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca08104690 0x7fca0819a0e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:36.774 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 >> 172.21.15.98:0/3603459192 conn(0x7fca08100060 msgr2=0x7fca08100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:36.774 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 shutdown_connections 2024-03-18T20:19:36.774 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:36.411+0000 7fca0e5a4700 1 -- 172.21.15.98:0/3603459192 wait complete. 2024-03-18T20:19:36.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:36 smithi098 ceph-mon[43462]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:36.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:36 smithi098 ceph-mon[43462]: Saving service mgr spec with placement count:2 2024-03-18T20:19:36.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:36 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:36.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:36 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3603459192' entity='client.admin' 2024-03-18T20:19:37.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:37 smithi098 ceph-mon[43462]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:19:37.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:37 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2130347759' entity='client.admin' 2024-03-18T20:19:37.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.153+0000 7faae1b69700 1 Processor -- start 2024-03-18T20:19:37.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.153+0000 7faae1b69700 1 -- start start 2024-03-18T20:19:37.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faae1b69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:37.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faae1b69700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc105060 con 0x7faadc104670 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47258/0 (socket says 172.21.15.98:47258) 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadb7fe700 1 -- 172.21.15.98:0/2208605209 learned_addr learned my addr 172.21.15.98:0/2208605209 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadb7fe700 1 -- 172.21.15.98:0/2208605209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc1051a0 con 0x7faadc104670 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadb7fe700 1 --2- 172.21.15.98:0/2208605209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faacc0098a0 tx=0x7faacc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1788cb4acd927e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.154+0000 7faadaffd700 1 -- 172.21.15.98:0/2208605209 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc004160 con 0x7faadc104670 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faadaffd700 1 -- 172.21.15.98:0/2208605209 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faacc0042c0 con 0x7faadc104670 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faadaffd700 1 -- 172.21.15.98:0/2208605209 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc0118e0 con 0x7faadc104670 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 -- 172.21.15.98:0/2208605209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 msgr2=0x7faadc104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 --2- 172.21.15.98:0/2208605209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faacc0098a0 tx=0x7faacc009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 -- 172.21.15.98:0/2208605209 shutdown_connections 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 --2- 172.21.15.98:0/2208605209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc104a90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 -- 172.21.15.98:0/2208605209 >> 172.21.15.98:0/2208605209 conn(0x7faadc100090 msgr2=0x7faadc1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 -- 172.21.15.98:0/2208605209 shutdown_connections 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.155+0000 7faae1b69700 1 -- 172.21.15.98:0/2208605209 wait complete. 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.156+0000 7faae1b69700 1 Processor -- start 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.156+0000 7faae1b69700 1 -- start start 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faae1b69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faae1b69700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc19a680 con 0x7faadc104670 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faadb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faadb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47274/0 (socket says 172.21.15.98:47274) 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faadb7fe700 1 -- 172.21.15.98:0/2130347759 learned_addr learned my addr 172.21.15.98:0/2130347759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faadb7fe700 1 -- 172.21.15.98:0/2130347759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc009550 con 0x7faadc104670 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faadb7fe700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faacc003f70 tx=0x7faacc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc0037d0 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faacc01b440 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc19a880 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc0193d0 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.157+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc19ad20 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.158+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7faacc019610 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.158+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc108310 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.159+0000 7faad97fa700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 0x7faac803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.159+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faacc04c350 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.159+0000 7faad2ffd700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 0x7faac803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.159+0000 7faad2ffd700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 0x7faac803abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faac4006fd0 tx=0x7faac4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.167+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faacc022360 con 0x7faadc104670 2024-03-18T20:19:37.922 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.378+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7faadc1051a0 con 0x7faadc104670 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.381+0000 7faad97fa700 1 -- 172.21.15.98:0/2130347759 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7faadc1051a0 con 0x7faadc104670 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 msgr2=0x7faac803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 0x7faac803abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faac4006fd0 tx=0x7faac4006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 msgr2=0x7faadc19a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faacc003f70 tx=0x7faacc0119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 shutdown_connections 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7faac8038700 0x7faac803abc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 --2- 172.21.15.98:0/2130347759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faadc104670 0x7faadc19a140 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.384+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 >> 172.21.15.98:0/2130347759 conn(0x7faadc100090 msgr2=0x7faadc100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.385+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 shutdown_connections 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:37.385+0000 7faae1b69700 1 -- 172.21.15.98:0/2130347759 wait complete. 2024-03-18T20:19:37.923 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-03-18T20:19:39.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1422491632' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T20:19:39.038 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:38 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: ignoring --setuser ceph since I am not root 2024-03-18T20:19:39.038 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:38 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: ignoring --setgroup ceph since I am not root 2024-03-18T20:19:39.203 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.249+0000 7f6a6e43f700 1 Processor -- start 2024-03-18T20:19:39.203 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.249+0000 7f6a6e43f700 1 -- start start 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a6e43f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a6e43f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68079e30 con 0x7f6a68106e80 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47282/0 (socket says 172.21.15.98:47282) 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a67fff700 1 -- 172.21.15.98:0/2164957117 learned_addr learned my addr 172.21.15.98:0/2164957117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a67fff700 1 -- 172.21.15.98:0/2164957117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a681097b0 con 0x7f6a68106e80 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a67fff700 1 --2- 172.21.15.98:0/2164957117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cefd21d75da3123 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a66ffd700 1 -- 172.21.15.98:0/2164957117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a58004160 con 0x7f6a68106e80 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a66ffd700 1 -- 172.21.15.98:0/2164957117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6a580042c0 con 0x7f6a68106e80 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.250+0000 7f6a66ffd700 1 -- 172.21.15.98:0/2164957117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a580118e0 con 0x7f6a68106e80 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 -- 172.21.15.98:0/2164957117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 msgr2=0x7f6a68109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/2164957117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a58009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 -- 172.21.15.98:0/2164957117 shutdown_connections 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/2164957117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a68109270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:39.204 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 -- 172.21.15.98:0/2164957117 >> 172.21.15.98:0/2164957117 conn(0x7f6a681008f0 msgr2=0x7f6a68102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 -- 172.21.15.98:0/2164957117 shutdown_connections 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.251+0000 7f6a6e43f700 1 -- 172.21.15.98:0/2164957117 wait complete. 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a6e43f700 1 Processor -- start 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a6e43f700 1 -- start start 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a6e43f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a6e43f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6819a620 con 0x7f6a68106e80 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47294/0 (socket says 172.21.15.98:47294) 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.252+0000 7f6a67fff700 1 -- 172.21.15.98:0/1422491632 learned_addr learned my addr 172.21.15.98:0/1422491632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a67fff700 1 -- 172.21.15.98:0/1422491632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58009550 con 0x7f6a68106e80 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a67fff700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6a58003e30 tx=0x7f6a58011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a580036a0 con 0x7f6a68106e80 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6a5801b440 con 0x7f6a68106e80 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6819a820 con 0x7f6a68106e80 2024-03-18T20:19:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a580193d0 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.253+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6819acc0 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f6a58019690 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6810a530 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a657fa700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 0x7f6a5003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6a5804c6a0 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a677fe700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 0x7f6a5003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.254+0000 7f6a677fe700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 0x7f6a5003ab70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006fd0 tx=0x7f6a5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.259+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a58018db0 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.505+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f6a6810a750 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.786+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f6a5801a750 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.787+0000 7f6a657fa700 1 -- 172.21.15.98:0/1422491632 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f6a5802b710 con 0x7f6a68106e80 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 msgr2=0x7f6a5003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 0x7f6a5003ab70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006fd0 tx=0x7f6a5c006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:39.206 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 msgr2=0x7f6a6819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6a58003e30 tx=0x7f6a58011e80 comp rx=0 tx=0).stop 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 shutdown_connections 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f6a500386b0 0x7f6a5003ab70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 --2- 172.21.15.98:0/1422491632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a68106e80 0x7f6a6819a0e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 >> 172.21.15.98:0/1422491632 conn(0x7f6a681008f0 msgr2=0x7f6a681014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 shutdown_connections 2024-03-18T20:19:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:38.789+0000 7f6a6e43f700 1 -- 172.21.15.98:0/1422491632 wait complete. 2024-03-18T20:19:39.289 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:39.230+0000 7f4b1e5b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:19:39.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:39.332+0000 7f4b1e5b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:19:39.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:39.422+0000 7f4b1e5b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:19:39.593 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:39.511+0000 7f4b1e5b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:19:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1422491632' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T20:19:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:39 smithi098 ceph-mon[43462]: mgrmap e9: x(active, since 13s) 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.544+0000 7fa569f7e700 1 Processor -- start 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.544+0000 7fa569f7e700 1 -- start start 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.544+0000 7fa569f7e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.544+0000 7fa569f7e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa564079e10 con 0x7fa564106ea0 2024-03-18T20:19:40.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47316/0 (socket says 172.21.15.98:47316) 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5637fe700 1 -- 172.21.15.98:0/3574450792 learned_addr learned my addr 172.21.15.98:0/3574450792 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5637fe700 1 -- 172.21.15.98:0/3574450792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5641097d0 con 0x7fa564106ea0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5637fe700 1 --2- 172.21.15.98:0/3574450792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098a0 tx=0x7fa554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e85f8d2210e67c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5627fc700 1 -- 172.21.15.98:0/3574450792 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa554004160 con 0x7fa564106ea0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5627fc700 1 -- 172.21.15.98:0/3574450792 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa5540042c0 con 0x7fa564106ea0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.545+0000 7fa5627fc700 1 -- 172.21.15.98:0/3574450792 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5540118e0 con 0x7fa564106ea0 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 -- 172.21.15.98:0/3574450792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 msgr2=0x7fa564109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 --2- 172.21.15.98:0/3574450792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098a0 tx=0x7fa554009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 -- 172.21.15.98:0/3574450792 shutdown_connections 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 --2- 172.21.15.98:0/3574450792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa564109290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 -- 172.21.15.98:0/3574450792 >> 172.21.15.98:0/3574450792 conn(0x7fa564100870 msgr2=0x7fa564102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 -- 172.21.15.98:0/3574450792 shutdown_connections 2024-03-18T20:19:40.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.546+0000 7fa569f7e700 1 -- 172.21.15.98:0/3574450792 wait complete. 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa569f7e700 1 Processor -- start 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa569f7e700 1 -- start start 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa569f7e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa569f7e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56419a600 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa5637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa5637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47326/0 (socket says 172.21.15.98:47326) 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.547+0000 7fa5637fe700 1 -- 172.21.15.98:0/2726802899 learned_addr learned my addr 172.21.15.98:0/2726802899 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa5637fe700 1 -- 172.21.15.98:0/2726802899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa554009550 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa5637fe700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5540046e0 tx=0x7fa5540119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5540037d0 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa55401b440 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa554023400 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56419a800 con 0x7fa564106ea0 2024-03-18T20:19:40.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.548+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56419aca0 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fa554023630 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56410a500 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa560ff9700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 0x7fa54c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa55404c5b0 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa562ffd700 1 -- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 msgr2=0x7fa54c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.549+0000 7fa562ffd700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 0x7fa54c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.554+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa554018350 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.750+0000 7fa562ffd700 1 -- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 msgr2=0x7fa54c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.750+0000 7fa562ffd700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 0x7fa54c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.805+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa56419dca0 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.806+0000 7fa560ff9700 1 -- 172.21.15.98:0/2726802899 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa55402bb40 con 0x7fa564106ea0 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 msgr2=0x7fa54c03ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:19:40.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 0x7fa54c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 msgr2=0x7fa56419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5540046e0 tx=0x7fa5540119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 shutdown_connections 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7fa54c0387a0 0x7fa54c03ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 --2- 172.21.15.98:0/2726802899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa564106ea0 0x7fa56419a0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 >> 172.21.15.98:0/2726802899 conn(0x7fa564100870 msgr2=0x7fa564101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 shutdown_connections 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:39.808+0000 7fa569f7e700 1 -- 172.21.15.98:0/2726802899 wait complete. 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-03-18T20:19:40.175 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-03-18T20:19:40.432 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.154+0000 7f4b1e5b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:40.432 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.427+0000 7f4b1e5b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:19:40.790 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.517+0000 7f4b1e5b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:19:40.790 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.628+0000 7f4b1e5b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:41.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2726802899' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T20:19:41.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.786+0000 7f4b1e5b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:19:41.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.912+0000 7f4b1e5b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:19:41.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:40.997+0000 7f4b1e5b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:19:41.755 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:41.365+0000 7f4b1e5b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:19:42.042 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:41.749+0000 7f4b1e5b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:19:42.042 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:41.944+0000 7f4b1e5b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:19:42.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:42 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:42.037+0000 7f4b1e5b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:19:43.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:42 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:42.789+0000 7f4b1e5b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:19:43.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:42 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:42.886+0000 7f4b1e5b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:19:43.093 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:43 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:43.032+0000 7f4b1e5b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:19:44.029 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:43 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:43.650+0000 7f4b1e5b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:19:44.029 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:43 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:43.831+0000 7f4b1e5b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:19:44.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:44 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:44.023+0000 7f4b1e5b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:19:45.225 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:44 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:44.938+0000 7f4b1e5b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:19:45.225 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:45.220+0000 7f4b1e5b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:19:45.502 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:45.308+0000 7f4b1e5b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:19:45.497+0000 7f4b1e5b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: Active manager daemon x restarted 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: Activating manager daemon x 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: mgrmap e10: x(active, starting, since 0.00332258s) 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:19:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:19:45.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:19:45.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: Manager daemon x is now available 2024-03-18T20:19:45.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:45.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.518+0000 7f415fdb9700 1 Processor -- start 2024-03-18T20:19:46.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.518+0000 7f415fdb9700 1 -- start start 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415fdb9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415fdb9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158105070 con 0x7f4158104680 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415db55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415db55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47336/0 (socket says 172.21.15.98:47336) 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415db55700 1 -- 172.21.15.98:0/1470856054 learned_addr learned my addr 172.21.15.98:0/1470856054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415db55700 1 -- 172.21.15.98:0/1470856054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41581051b0 con 0x7f4158104680 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415db55700 1 --2- 172.21.15.98:0/1470856054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41540098a0 tx=0x7f4154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c6ac08e177b222 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415cb53700 1 -- 172.21.15.98:0/1470856054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4154004160 con 0x7f4158104680 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415cb53700 1 -- 172.21.15.98:0/1470856054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f41540042c0 con 0x7f4158104680 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.519+0000 7f415cb53700 1 -- 172.21.15.98:0/1470856054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41540118e0 con 0x7f4158104680 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 -- 172.21.15.98:0/1470856054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 msgr2=0x7f4158104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 --2- 172.21.15.98:0/1470856054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41540098a0 tx=0x7f4154009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 -- 172.21.15.98:0/1470856054 shutdown_connections 2024-03-18T20:19:46.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 --2- 172.21.15.98:0/1470856054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f4158104aa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 -- 172.21.15.98:0/1470856054 >> 172.21.15.98:0/1470856054 conn(0x7f4158100060 msgr2=0x7f4158102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 -- 172.21.15.98:0/1470856054 shutdown_connections 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.520+0000 7f415fdb9700 1 -- 172.21.15.98:0/1470856054 wait complete. 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415fdb9700 1 Processor -- start 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415fdb9700 1 -- start start 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415fdb9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415fdb9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415819a590 con 0x7f4158104680 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415db55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415db55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47352/0 (socket says 172.21.15.98:47352) 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415db55700 1 -- 172.21.15.98:0/2258193432 learned_addr learned my addr 172.21.15.98:0/2258193432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.521+0000 7f415db55700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154009550 con 0x7f4158104680 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f415db55700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4154003f70 tx=0x7f41540119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41540036a0 con 0x7f4158104680 2024-03-18T20:19:46.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f415401b440 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415819a7f0 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4154023400 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.522+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415819d4c0 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f415401b5b0 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f414affd700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f414403b320 con 0x7f4144038750 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f415404c640 con 0x7f4158104680 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f415d354700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.523+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.724+0000 7f415d354700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:40.724+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:41.125+0000 7f415d354700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:41.125+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:19:46.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:41.926+0000 7f415d354700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:41.926+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:43.527+0000 7f415d354700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1287611061 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:43.527+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:45.513+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) v1 ==== 45313+0+0 (secure 0 0 0) 0x7f415401acd0 con 0x7f4158104680 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:45.513+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 msgr2=0x7f414403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:45.513+0000 7f414affd700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.517+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7f4154003800 con 0x7f4158104680 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.517+0000 7f414affd700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 0x7f414403e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.517+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f414403b320 con 0x7f414403bff0 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.517+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 0x7f414403e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.518+0000 7f415d354700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 0x7f414403e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f414c003a10 tx=0x7f414c0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.518+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f414403b320 con 0x7f414403bff0 2024-03-18T20:19:46.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f415810a930 con 0x7f414403bff0 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f414affd700 1 -- 172.21.15.98:0/2258193432 <== mgr.14152 v2:172.21.15.98:6800/3114713517 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f414c00a050 con 0x7f414403bff0 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 msgr2=0x7f414403e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 0x7f414403e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f414c003a10 tx=0x7f414c0092b0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 msgr2=0x7f415819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4154003f70 tx=0x7f41540119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 shutdown_connections 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f414403bff0 0x7f414403e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4158104680 0x7f415819a050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 --2- 172.21.15.98:0/2258193432 >> [v2:172.21.15.98:6800/1287611061,v1:172.21.15.98:6801/1287611061] conn(0x7f4144038750 0x7f414403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 >> 172.21.15.98:0/2258193432 conn(0x7f4158100060 msgr2=0x7f4158100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 shutdown_connections 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:46.525+0000 7f415fdb9700 1 -- 172.21.15.98:0/2258193432 wait complete. 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-03-18T20:19:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-03-18T20:19:46.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:19:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:19:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:45] ENGINE Bus STARTING 2024-03-18T20:19:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:45] ENGINE Serving on https://172.21.15.98:7150 2024-03-18T20:19:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:46 smithi098 ceph-mon[43462]: mgrmap e11: x(active, since 1.00804s) 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:45] ENGINE Serving on http://172.21.15.98:8765 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: [18/Mar/2024:20:19:45] ENGINE Bus STARTED 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:47.837 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:48.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e9617700 1 Processor -- start 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e9617700 1 -- start start 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e9617700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e9617700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4106de0 con 0x7f47e41063f0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59140/0 (socket says 172.21.15.98:59140) 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e2ffd700 1 -- 172.21.15.98:0/3720905580 learned_addr learned my addr 172.21.15.98:0/3720905580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.229+0000 7f47e2ffd700 1 -- 172.21.15.98:0/3720905580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e41075a0 con 0x7f47e41063f0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e2ffd700 1 --2- 172.21.15.98:0/3720905580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47d40098a0 tx=0x7f47d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436f46786f3b1274 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e1ffb700 1 -- 172.21.15.98:0/3720905580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d4004160 con 0x7f47e41063f0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e1ffb700 1 -- 172.21.15.98:0/3720905580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47d40042c0 con 0x7f47e41063f0 2024-03-18T20:19:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e1ffb700 1 -- 172.21.15.98:0/3720905580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d40118e0 con 0x7f47e41063f0 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 -- 172.21.15.98:0/3720905580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 msgr2=0x7f47e4106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 --2- 172.21.15.98:0/3720905580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47d40098a0 tx=0x7f47d4009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 -- 172.21.15.98:0/3720905580 shutdown_connections 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 --2- 172.21.15.98:0/3720905580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e4106810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 -- 172.21.15.98:0/3720905580 >> 172.21.15.98:0/3720905580 conn(0x7f47e41019b0 msgr2=0x7f47e4103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.230+0000 7f47e9617700 1 -- 172.21.15.98:0/3720905580 shutdown_connections 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.231+0000 7f47e9617700 1 -- 172.21.15.98:0/3720905580 wait complete. 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.231+0000 7f47e9617700 1 Processor -- start 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.231+0000 7f47e9617700 1 -- start start 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.231+0000 7f47e9617700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.231+0000 7f47e9617700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e407c040 con 0x7f47e41063f0 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59146/0 (socket says 172.21.15.98:59146) 2024-03-18T20:19:48.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e2ffd700 1 -- 172.21.15.98:0/1263481466 learned_addr learned my addr 172.21.15.98:0/1263481466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e2ffd700 1 -- 172.21.15.98:0/1263481466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4009550 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e2ffd700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47d4003e30 tx=0x7f47d4011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d40036a0 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e407a280 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47d401b440 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d40193d0 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.232+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e407a720 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.233+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7f47d4019690 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.233+0000 7f47cbfff700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 0x7f47cc03aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.233+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f47d404c420 con 0x7f47e41063f0 2024-03-18T20:19:48.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.233+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e404f2d0 con 0x7f47e41063f0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.233+0000 7f47e27fc700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 0x7f47cc03aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.234+0000 7f47e27fc700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 0x7f47cc03aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f47d8006fd0 tx=0x7f47d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.238+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f47d401a3d0 con 0x7f47e41063f0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.451+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f47e407af70 con 0x7f47cc0385c0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.578+0000 7f47cbfff700 1 -- 172.21.15.98:0/1263481466 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f47e407af70 con 0x7f47cc0385c0 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 msgr2=0x7f47cc03aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 0x7f47cc03aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f47d8006fd0 tx=0x7f47d8006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 msgr2=0x7f47e407bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47d4003e30 tx=0x7f47d4011e80 comp rx=0 tx=0).stop 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 shutdown_connections 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47cc0385c0 0x7f47cc03aa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 --2- 172.21.15.98:0/1263481466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e41063f0 0x7f47e407bb00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 >> 172.21.15.98:0/1263481466 conn(0x7f47e41019b0 msgr2=0x7f47e41025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 shutdown_connections 2024-03-18T20:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:47.580+0000 7f47e9617700 1 -- 172.21.15.98:0/1263481466 wait complete. 2024-03-18T20:19:48.174 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-03-18T20:19:48.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:48 smithi098 ceph-mon[43462]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:48.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:48.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:48 smithi098 ceph-mon[43462]: mgrmap e12: x(active, since 2s) 2024-03-18T20:19:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$aGxSiG1cvSw5SqpQgUNIf.DohwWva.WOeGw3yKYRV5QU3111OY1OO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710793188, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T20:19:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.510+0000 7f0efe854700 1 Processor -- start 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.510+0000 7f0efe854700 1 -- start start 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.510+0000 7f0efe854700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.510+0000 7f0efe854700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8079e90 con 0x7f0ef8106e20 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59148/0 (socket says 172.21.15.98:59148) 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef7fff700 1 -- 172.21.15.98:0/895127469 learned_addr learned my addr 172.21.15.98:0/895127469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef7fff700 1 -- 172.21.15.98:0/895127469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef8109750 con 0x7f0ef8106e20 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef7fff700 1 --2- 172.21.15.98:0/895127469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80098a0 tx=0x7f0ee8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea6144eb868680e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef6ffd700 1 -- 172.21.15.98:0/895127469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee8004160 con 0x7f0ef8106e20 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef6ffd700 1 -- 172.21.15.98:0/895127469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ee80042c0 con 0x7f0ef8106e20 2024-03-18T20:19:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.511+0000 7f0ef6ffd700 1 -- 172.21.15.98:0/895127469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee80118e0 con 0x7f0ef8106e20 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- 172.21.15.98:0/895127469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 msgr2=0x7f0ef8109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 --2- 172.21.15.98:0/895127469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80098a0 tx=0x7f0ee8009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- 172.21.15.98:0/895127469 shutdown_connections 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 --2- 172.21.15.98:0/895127469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8109210 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- 172.21.15.98:0/895127469 >> 172.21.15.98:0/895127469 conn(0x7f0ef8100870 msgr2=0x7f0ef8102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- 172.21.15.98:0/895127469 shutdown_connections 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- 172.21.15.98:0/895127469 wait complete. 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 Processor -- start 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.512+0000 7f0efe854700 1 -- start start 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0efe854700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0efe854700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8106b90 con 0x7f0ef8106e20 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59162/0 (socket says 172.21.15.98:59162) 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef7fff700 1 -- 172.21.15.98:0/2860739900 learned_addr learned my addr 172.21.15.98:0/2860739900 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef7fff700 1 -- 172.21.15.98:0/2860739900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009550 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef7fff700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8003e30 tx=0x7f0ee8011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee80036a0 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ee801b440 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee80193d0 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.513+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef8104dd0 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.514+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef8105270 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.514+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f0ee8003800 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.515+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef810a530 con 0x7f0ef8106e20 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.515+0000 7f0ef57fa700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 0x7f0ee003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.515+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ee804c670 con 0x7f0ef8106e20 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.515+0000 7f0ef77fe700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 0x7f0ee003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.515+0000 7f0ef77fe700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 0x7f0ee003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0eec006fd0 tx=0x7f0eec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.520+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ee802b7c0 con 0x7f0ef8106e20 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.728+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0ef8062200 con 0x7f0ee0038700 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.947+0000 7f0ef57fa700 1 -- 172.21.15.98:0/2860739900 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f0ef8062200 con 0x7f0ee0038700 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.949+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 msgr2=0x7f0ee003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.949+0000 7f0efe854700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 0x7f0ee003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0eec006fd0 tx=0x7f0eec006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.949+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 msgr2=0x7f0ef8106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.949+0000 7f0efe854700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8003e30 tx=0x7f0ee8011e80 comp rx=0 tx=0).stop 2024-03-18T20:19:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 shutdown_connections 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0ee0038700 0x7f0ee003abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 --2- 172.21.15.98:0/2860739900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef8106e20 0x7f0ef8106650 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 >> 172.21.15.98:0/2860739900 conn(0x7f0ef8100870 msgr2=0x7f0ef81013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 shutdown_connections 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:48.950+0000 7f0efe854700 1 -- 172.21.15.98:0/2860739900 wait complete. 2024-03-18T20:19:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-03-18T20:19:49.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:49 smithi098 ceph-mon[43462]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:49.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a415fb700 1 Processor -- start 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a415fb700 1 -- start start 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a415fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a415fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c079e50 con 0x7f7a3c106e60 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59166/0 (socket says 172.21.15.98:59166) 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.658+0000 7f7a3affd700 1 -- 172.21.15.98:0/2452384543 learned_addr learned my addr 172.21.15.98:0/2452384543 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.659+0000 7f7a3affd700 1 -- 172.21.15.98:0/2452384543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c109790 con 0x7f7a3c106e60 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.659+0000 7f7a3affd700 1 --2- 172.21.15.98:0/2452384543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0098a0 tx=0x7f7a2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148652e72b40827a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.659+0000 7f7a39ffb700 1 -- 172.21.15.98:0/2452384543 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a2c004160 con 0x7f7a3c106e60 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.659+0000 7f7a39ffb700 1 -- 172.21.15.98:0/2452384543 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a2c0042c0 con 0x7f7a3c106e60 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.659+0000 7f7a39ffb700 1 -- 172.21.15.98:0/2452384543 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a2c0118e0 con 0x7f7a3c106e60 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 -- 172.21.15.98:0/2452384543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 msgr2=0x7f7a3c109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 --2- 172.21.15.98:0/2452384543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0098a0 tx=0x7f7a2c009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 -- 172.21.15.98:0/2452384543 shutdown_connections 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 --2- 172.21.15.98:0/2452384543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c109250 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 -- 172.21.15.98:0/2452384543 >> 172.21.15.98:0/2452384543 conn(0x7f7a3c1008f0 msgr2=0x7f7a3c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 -- 172.21.15.98:0/2452384543 shutdown_connections 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 -- 172.21.15.98:0/2452384543 wait complete. 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.660+0000 7f7a415fb700 1 Processor -- start 2024-03-18T20:19:50.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a415fb700 1 -- start start 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a415fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a415fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c19a610 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59168/0 (socket says 172.21.15.98:59168) 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a3affd700 1 -- 172.21.15.98:0/913871302 learned_addr learned my addr 172.21.15.98:0/913871302 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a3affd700 1 -- 172.21.15.98:0/913871302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c009550 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a3affd700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c003e30 tx=0x7f7a2c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a2c0036a0 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a2c01b440 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c19a810 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a2c0193d0 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.661+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c19acb0 con 0x7f7a3c106e60 2024-03-18T20:19:50.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.662+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f7a2c019640 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.662+0000 7f7a23fff700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 0x7f7a2403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.662+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7a2c04c670 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.662+0000 7f7a3a7fc700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 0x7f7a2403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.662+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c04f2d0 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.663+0000 7f7a3a7fc700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 0x7f7a2403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006fd0 tx=0x7f7a30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.667+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a2c01a3d0 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.857+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7a3c19d690 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.858+0000 7f7a23fff700 1 -- 172.21.15.98:0/913871302 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7a2c04a020 con 0x7f7a3c106e60 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 msgr2=0x7f7a2403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 0x7f7a2403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006fd0 tx=0x7f7a30006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 msgr2=0x7f7a3c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c003e30 tx=0x7f7a2c011e80 comp rx=0 tx=0).stop 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 shutdown_connections 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7a24038750 0x7f7a2403ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:50.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 --2- 172.21.15.98:0/913871302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a3c106e60 0x7f7a3c19a0d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:50.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 >> 172.21.15.98:0/913871302 conn(0x7f7a3c1008f0 msgr2=0x7f7a3c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:50.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 shutdown_connections 2024-03-18T20:19:50.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:49.860+0000 7f7a415fb700 1 -- 172.21.15.98:0/913871302 wait complete. 2024-03-18T20:19:50.216 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T20:19:50.216 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-18T20:19:50.223 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T20:19:50.223 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-18T20:19:50.223 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-18T20:19:50.223 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout: Password: f0oajvx7uo 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:50.225 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config directory 2024-03-18T20:19:50.226 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-03-18T20:19:50.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:50 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/913871302' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T20:19:51.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a89db8700 1 Processor -- start 2024-03-18T20:19:51.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a89db8700 1 -- start start 2024-03-18T20:19:51.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a89db8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a89db8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84079e50 con 0x7f4a84106e60 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59180/0 (socket says 172.21.15.98:59180) 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a837fe700 1 -- 172.21.15.98:0/3468320501 learned_addr learned my addr 172.21.15.98:0/3468320501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a837fe700 1 -- 172.21.15.98:0/3468320501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84109790 con 0x7f4a84106e60 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.550+0000 7f4a837fe700 1 --2- 172.21.15.98:0/3468320501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0098a0 tx=0x7f4a6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6310c17135d66b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a827fc700 1 -- 172.21.15.98:0/3468320501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a6c004160 con 0x7f4a84106e60 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a827fc700 1 -- 172.21.15.98:0/3468320501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a6c0042c0 con 0x7f4a84106e60 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a827fc700 1 -- 172.21.15.98:0/3468320501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a6c0118e0 con 0x7f4a84106e60 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 -- 172.21.15.98:0/3468320501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 msgr2=0x7f4a84109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3468320501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0098a0 tx=0x7f4a6c009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 -- 172.21.15.98:0/3468320501 shutdown_connections 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3468320501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a84109250 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 -- 172.21.15.98:0/3468320501 >> 172.21.15.98:0/3468320501 conn(0x7f4a841008f0 msgr2=0x7f4a84102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 -- 172.21.15.98:0/3468320501 shutdown_connections 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.551+0000 7f4a89db8700 1 -- 172.21.15.98:0/3468320501 wait complete. 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a89db8700 1 Processor -- start 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a89db8700 1 -- start start 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a89db8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a89db8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419a610 con 0x7f4a84106e60 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59190/0 (socket says 172.21.15.98:59190) 2024-03-18T20:19:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a837fe700 1 -- 172.21.15.98:0/3412139962 learned_addr learned my addr 172.21.15.98:0/3412139962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.552+0000 7f4a837fe700 1 -- 172.21.15.98:0/3412139962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c009550 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a837fe700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0046e0 tx=0x7f4a6c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a6c0037d0 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8419a810 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a6c01b440 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a6c023400 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.553+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8419acb0 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.554+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8410a510 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.554+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f4a6c01b840 con 0x7f4a84106e60 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.554+0000 7f4a80ff9700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 0x7f4a7003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:51.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.554+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4a6c04c6e0 con 0x7f4a84106e60 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.554+0000 7f4a82ffd700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 0x7f4a7003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.555+0000 7f4a82ffd700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 0x7f4a7003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006fd0 tx=0x7f4a74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.559+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a6c019da0 con 0x7f4a84106e60 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.760+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4a8410a700 con 0x7f4a84106e60 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.761+0000 7f4a80ff9700 1 -- 172.21.15.98:0/3412139962 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4a6c019bc0 con 0x7f4a84106e60 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 msgr2=0x7f4a7003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 0x7f4a7003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006fd0 tx=0x7f4a74006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 msgr2=0x7f4a8419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0046e0 tx=0x7f4a6c0119a0 comp rx=0 tx=0).stop 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 shutdown_connections 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a70038750 0x7f4a7003ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:51.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 --2- 172.21.15.98:0/3412139962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a84106e60 0x7f4a8419a0d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:51.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 >> 172.21.15.98:0/3412139962 conn(0x7f4a841008f0 msgr2=0x7f4a841014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:51.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 shutdown_connections 2024-03-18T20:19:51.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:50.763+0000 7f4a89db8700 1 -- 172.21.15.98:0/3412139962 wait complete. 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.465+0000 7f7132ff4700 1 Processor -- start 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.465+0000 7f7132ff4700 1 -- start start 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7132ff4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7132ff4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c079e30 con 0x7f712c106e90 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7130d90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7130d90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59198/0 (socket says 172.21.15.98:59198) 2024-03-18T20:19:52.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7130d90700 1 -- 172.21.15.98:0/639528300 learned_addr learned my addr 172.21.15.98:0/639528300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7130d90700 1 -- 172.21.15.98:0/639528300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c1097c0 con 0x7f712c106e90 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f7130d90700 1 --2- 172.21.15.98:0/639528300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ae243c0ee3df5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f712b7fe700 1 -- 172.21.15.98:0/639528300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7120004160 con 0x7f712c106e90 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f712b7fe700 1 -- 172.21.15.98:0/639528300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f71200042c0 con 0x7f712c106e90 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.466+0000 7f712b7fe700 1 -- 172.21.15.98:0/639528300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71200118e0 con 0x7f712c106e90 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 -- 172.21.15.98:0/639528300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 msgr2=0x7f712c109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 --2- 172.21.15.98:0/639528300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).stop 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 -- 172.21.15.98:0/639528300 shutdown_connections 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 --2- 172.21.15.98:0/639528300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c109280 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 -- 172.21.15.98:0/639528300 >> 172.21.15.98:0/639528300 conn(0x7f712c1008a0 msgr2=0x7f712c102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 -- 172.21.15.98:0/639528300 shutdown_connections 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.467+0000 7f7132ff4700 1 -- 172.21.15.98:0/639528300 wait complete. 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7132ff4700 1 Processor -- start 2024-03-18T20:19:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7132ff4700 1 -- start start 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7132ff4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7132ff4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c19a610 con 0x7f712c106e90 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7130d90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7130d90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59202/0 (socket says 172.21.15.98:59202) 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7130d90700 1 -- 172.21.15.98:0/754781846 learned_addr learned my addr 172.21.15.98:0/754781846 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7130d90700 1 -- 172.21.15.98:0/754781846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120009550 con 0x7f712c106e90 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7130d90700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7120006af0 tx=0x7f7120011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71200037d0 con 0x7f712c106e90 2024-03-18T20:19:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f712001b440 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c19a810 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.468+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7120023400 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.469+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c19acb0 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.469+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f71200236c0 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.469+0000 7f7129ffb700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 0x7f711c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.470+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7120003b30 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.470+0000 7f712bfff700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 0x7f711c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.470+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c04f2d0 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.470+0000 7f712bfff700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 0x7f711c03abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7118006fd0 tx=0x7f7118006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.475+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f71200239c0 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.775+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f712c19d4f0 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.780+0000 7f7129ffb700 1 -- 172.21.15.98:0/754781846 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f712004a020 con 0x7f712c106e90 2024-03-18T20:19:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.782+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 msgr2=0x7f711c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.782+0000 7f7132ff4700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 0x7f711c03abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7118006fd0 tx=0x7f7118006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 msgr2=0x7f712c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7120006af0 tx=0x7f7120011ed0 comp rx=0 tx=0).stop 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 shutdown_connections 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f711c038700 0x7f711c03abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 --2- 172.21.15.98:0/754781846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f712c106e90 0x7f712c19a0d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 >> 172.21.15.98:0/754781846 conn(0x7f712c1008a0 msgr2=0x7f712c101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 shutdown_connections 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-18T20:19:51.783+0000 7f7132ff4700 1 -- 172.21.15.98:0/754781846 wait complete. 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:19:52.173 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-03-18T20:19:52.201 INFO:tasks.cephadm:Fetching config... 2024-03-18T20:19:52.202 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:19:52.202 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T20:19:52.218 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T20:19:52.218 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:19:52.219 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T20:19:52.278 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T20:19:52.278 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:19:52.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T20:19:52.352 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T20:19:52.352 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:19:52.352 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T20:19:52.409 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T20:19:52.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T20:19:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:52.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T20:19:52.544 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:52.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T20:19:52.603 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBDCA819ij3hlUF0ydM5qq8BIaBBGHYMTAZnbedLDxZJjPOX8kSOeyGGjvkU+S2iDPrMyBjvwnKrdFP3n/eqz48OJX6ZM2PUOgtiY2gMJJJEDEjkQlR3srnY9RzELO1NyvFG7l84JQXEYkkx06JrM7TXgwZhT+D5eb1OMWidq3lwPaPtnjS8D6nFMxmuCkTpMRzTLQPzwT+dsRLUY3TvWuAxrchumSdjx2XyB8h9pKP0GaoK8zgx5URzE+oPKdJF1UgCo6dJEwx3FgrS5/NWnFM5KxcGdAsyRyksioqwEe1FnkcL6LY148RDmtuuBbSA1oKq4R9L0VW1EHsFa/XSDgU2gDJxF0dcVXsDJkg5jnujQdzimzjA3TSOcToGPtdV+bmuj+POv915FxOB97HMgql6qvWs3tT1NjbrrOC5aNyuOk83TXgRBWaNrhtcvZ9kTQFlVhWiY0NdHU+WYxt9FRjMTMrQjEVx95ZCt9CbandH78B4zSdxZpslRIWgjagM= ceph-a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:19:52.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T20:19:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:19:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:52 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/754781846' entity='client.admin' 2024-03-18T20:19:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:52 smithi098 ceph-mon[43462]: mgrmap e13: x(active, since 6s) 2024-03-18T20:19:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- 172.21.15.98:0/2831761648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 msgr2=0x7f6a8c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 --2- 172.21.15.98:0/2831761648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c0ff460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009910 tx=0x7f6a7c009c20 comp rx=0 tx=0).stop 2024-03-18T20:19:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- 172.21.15.98:0/2831761648 shutdown_connections 2024-03-18T20:19:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 --2- 172.21.15.98:0/2831761648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c0ff460 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- 172.21.15.98:0/2831761648 >> 172.21.15.98:0/2831761648 conn(0x7f6a8c0fabe0 msgr2=0x7f6a8c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- 172.21.15.98:0/2831761648 shutdown_connections 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- 172.21.15.98:0/2831761648 wait complete. 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 Processor -- start 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.924+0000 7f6a93c19700 1 -- start start 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a93c19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a93c19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c1981d0 con 0x7f6a8c0ff040 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a919b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a919b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50518/0 (socket says 172.21.15.98:50518) 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a919b5700 1 -- 172.21.15.98:0/3995447124 learned_addr learned my addr 172.21.15.98:0/3995447124 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a919b5700 1 -- 172.21.15.98:0/3995447124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c0095f0 con 0x7f6a8c0ff040 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a919b5700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c005950 tx=0x7f6a7c005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a7c00b960 con 0x7f6a8c0ff040 2024-03-18T20:19:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6a7c00f460 con 0x7f6a8c0ff040 2024-03-18T20:19:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c1983d0 con 0x7f6a8c0ff040 2024-03-18T20:19:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a7c017440 con 0x7f6a8c0ff040 2024-03-18T20:19:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.925+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c198870 con 0x7f6a8c0ff040 2024-03-18T20:19:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.926+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f6a7c00bac0 con 0x7f6a8c0ff040 2024-03-18T20:19:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.926+0000 7f6a82ffd700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 0x7f6a7803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.926+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c191500 con 0x7f6a8c0ff040 2024-03-18T20:19:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.926+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6a7c04c800 con 0x7f6a8c0ff040 2024-03-18T20:19:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.926+0000 7f6a911b4700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 0x7f6a7803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.927+0000 7f6a911b4700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 0x7f6a7803acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a88006fd0 tx=0x7f6a88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:53.931+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a7c0292c0 con 0x7f6a8c0ff040 2024-03-18T20:19:54.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.141+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6a8c02cf60 con 0x7f6a8c0ff040 2024-03-18T20:19:54.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.144+0000 7f6a82ffd700 1 -- 172.21.15.98:0/3995447124 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6a7c0292c0 con 0x7f6a8c0ff040 2024-03-18T20:19:54.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 msgr2=0x7f6a7803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:54.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 0x7f6a7803acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a88006fd0 tx=0x7f6a88006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 msgr2=0x7f6a8c197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c005950 tx=0x7f6a7c005810 comp rx=0 tx=0).stop 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 shutdown_connections 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6a780387f0 0x7f6a7803acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 --2- 172.21.15.98:0/3995447124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a8c0ff040 0x7f6a8c197c90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.146+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 >> 172.21.15.98:0/3995447124 conn(0x7f6a8c0fabe0 msgr2=0x7f6a8c0fb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.147+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 shutdown_connections 2024-03-18T20:19:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:54.147+0000 7f6a93c19700 1 -- 172.21.15.98:0/3995447124 wait complete. 2024-03-18T20:19:54.516 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T20:19:54.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T20:19:54.742 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:19:55.406 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:55 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3995447124' entity='client.admin' 2024-03-18T20:19:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/2921157651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 msgr2=0x7f1c580ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/2921157651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c580ffa60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009910 tx=0x7f1c4c009c20 comp rx=0 tx=0).stop 2024-03-18T20:19:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/2921157651 shutdown_connections 2024-03-18T20:19:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/2921157651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c580ffa60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/2921157651 >> 172.21.15.98:0/2921157651 conn(0x7f1c580fabe0 msgr2=0x7f1c580fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/2921157651 shutdown_connections 2024-03-18T20:19:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.753+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/2921157651 wait complete. 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5cc4c700 1 Processor -- start 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5cc4c700 1 -- start start 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5cc4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5cc4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5818c9e0 con 0x7f1c580ff640 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50532/0 (socket says 172.21.15.98:50532) 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5659c700 1 -- 172.21.15.98:0/1716967855 learned_addr learned my addr 172.21.15.98:0/1716967855 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.754+0000 7f1c5659c700 1 -- 172.21.15.98:0/1716967855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c0095f0 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c5659c700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0098e0 tx=0x7f1c4c005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c4c00ba90 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5818ac20 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1c4c00bbf0 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c4c00fe70 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.755+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5818b0c0 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f1c4c00f460 con 0x7f1c580ff640 2024-03-18T20:19:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c477fe700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 0x7f1c4003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c580623e0 con 0x7f1c580ff640 2024-03-18T20:19:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c4c04c8e0 con 0x7f1c580ff640 2024-03-18T20:19:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c55d9b700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 0x7f1c4003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.756+0000 7f1c55d9b700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 0x7f1c4003ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c48006fd0 tx=0x7f1c48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.761+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1c4c02ad30 con 0x7f1c580ff640 2024-03-18T20:19:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.957+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c5818bab0 con 0x7f1c400387a0 2024-03-18T20:19:55.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.961+0000 7f1c477fe700 1 -- 172.21.15.98:0/1716967855 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1c5818bab0 con 0x7f1c400387a0 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 msgr2=0x7f1c4003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 0x7f1c4003ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c48006fd0 tx=0x7f1c48006e40 comp rx=0 tx=0).stop 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 msgr2=0x7f1c5818c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0098e0 tx=0x7f1c4c005810 comp rx=0 tx=0).stop 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 shutdown_connections 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1c400387a0 0x7f1c4003ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 --2- 172.21.15.98:0/1716967855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c580ff640 0x7f1c5818c4a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 >> 172.21.15.98:0/1716967855 conn(0x7f1c580fabe0 msgr2=0x7f1c58068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 shutdown_connections 2024-03-18T20:19:55.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:55.963+0000 7f1c5cc4c700 1 -- 172.21.15.98:0/1716967855 wait complete. 2024-03-18T20:19:56.379 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-03-18T20:19:56.380 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:19:56.380 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-03-18T20:19:56.401 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:19:56.401 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:19:56.460 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-03-18T20:19:56.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch host add smithi136 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:19:56.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:19:56.688 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:19:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 -- 172.21.15.98:0/1867803270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 msgr2=0x7f8228106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 --2- 172.21.15.98:0/1867803270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f8228106ab0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8218009910 tx=0x7f8218009c20 comp rx=0 tx=0).stop 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 -- 172.21.15.98:0/1867803270 shutdown_connections 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 --2- 172.21.15.98:0/1867803270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f8228106ab0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 -- 172.21.15.98:0/1867803270 >> 172.21.15.98:0/1867803270 conn(0x7f8228075110 msgr2=0x7f8228075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 -- 172.21.15.98:0/1867803270 shutdown_connections 2024-03-18T20:19:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.683+0000 7f822d195700 1 -- 172.21.15.98:0/1867803270 wait complete. 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.684+0000 7f822d195700 1 Processor -- start 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.684+0000 7f822d195700 1 -- start start 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f822d195700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f822d195700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82281015a0 con 0x7f82281066d0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f8226d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f8226d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50546/0 (socket says 172.21.15.98:50546) 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f8226d9d700 1 -- 172.21.15.98:0/2917293565 learned_addr learned my addr 172.21.15.98:0/2917293565 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f8226d9d700 1 -- 172.21.15.98:0/2917293565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180095f0 con 0x7f82281066d0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f8226d9d700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8218005660 tx=0x7f8218005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f821801d070 con 0x7f82281066d0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f821800b6f0 con 0x7f82281066d0 2024-03-18T20:19:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82280ffc20 con 0x7f82281066d0 2024-03-18T20:19:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82180037f0 con 0x7f82281066d0 2024-03-18T20:19:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.685+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82281000a0 con 0x7f82281066d0 2024-03-18T20:19:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.686+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f8218003ab0 con 0x7f82281066d0 2024-03-18T20:19:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.686+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8228101e10 con 0x7f82281066d0 2024-03-18T20:19:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.686+0000 7f820ffff700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 0x7f821003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:19:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.686+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f821804c430 con 0x7f82281066d0 2024-03-18T20:19:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.687+0000 7f822659c700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 0x7f821003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:19:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.687+0000 7f822659c700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 0x7f821003ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f821c006fd0 tx=0x7f821c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:19:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.691+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f821802ad80 con 0x7f82281066d0 2024-03-18T20:19:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:19:57.888+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}) v1 -- 0x7f82281002e0 con 0x7f82100387a0 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: Updating smithi098:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: Updating smithi098:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:57 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:19:59.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:58 smithi098 ceph-mon[43462]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:19:59.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:19:58 smithi098 ceph-mon[43462]: Deploying cephadm binary to smithi136 2024-03-18T20:20:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.009+0000 7f820ffff700 1 -- 172.21.15.98:0/2917293565 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f82281002e0 con 0x7f82100387a0 2024-03-18T20:20:00.014 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-03-18T20:20:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.011+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 msgr2=0x7f821003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.011+0000 7f822d195700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 0x7f821003ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f821c006fd0 tx=0x7f821c006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.011+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 msgr2=0x7f82280ff6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.011+0000 7f822d195700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8218005660 tx=0x7f8218005740 comp rx=0 tx=0).stop 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 shutdown_connections 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f82100387a0 0x7f821003ac60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 --2- 172.21.15.98:0/2917293565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82281066d0 0x7f82280ff6e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 >> 172.21.15.98:0/2917293565 conn(0x7f8228075110 msgr2=0x7f82280fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 shutdown_connections 2024-03-18T20:20:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:00.012+0000 7f822d195700 1 -- 172.21.15.98:0/2917293565 wait complete. 2024-03-18T20:20:00.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T20:20:00.607 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:01 smithi098 ceph-mon[43462]: Added host smithi136 2024-03-18T20:20:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 -- 172.21.15.98:0/3435223739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 msgr2=0x7f32d0105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 --2- 172.21.15.98:0/3435223739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0105a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009910 tx=0x7f32c0009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 -- 172.21.15.98:0/3435223739 shutdown_connections 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 --2- 172.21.15.98:0/3435223739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0105a10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 -- 172.21.15.98:0/3435223739 >> 172.21.15.98:0/3435223739 conn(0x7f32d0075110 msgr2=0x7f32d0075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 -- 172.21.15.98:0/3435223739 shutdown_connections 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.557+0000 7f32d6433700 1 -- 172.21.15.98:0/3435223739 wait complete. 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32d6433700 1 Processor -- start 2024-03-18T20:20:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32d6433700 1 -- start start 2024-03-18T20:20:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32d6433700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32d6433700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d0198540 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50568/0 (socket says 172.21.15.98:50568) 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32cffff700 1 -- 172.21.15.98:0/111363920 learned_addr learned my addr 172.21.15.98:0/111363920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32cffff700 1 -- 172.21.15.98:0/111363920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00095f0 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.558+0000 7f32cffff700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005660 tx=0x7f32c0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32c001d070 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0198740 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f32c000b750 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32c00037d0 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d0198be0 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f32c0003a90 con 0x7f32d0105630 2024-03-18T20:20:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.559+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d0100d00 con 0x7f32d0105630 2024-03-18T20:20:01.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.560+0000 7f32cd7fa700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 0x7f32b803ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:01.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.560+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f32c004c490 con 0x7f32d0105630 2024-03-18T20:20:01.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.560+0000 7f32cf7fe700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 0x7f32b803ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:01.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.560+0000 7f32cf7fe700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 0x7f32b803ac50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006fd0 tx=0x7f32c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.564+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32c0029770 con 0x7f32d0105630 2024-03-18T20:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.761+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32d00fec30 con 0x7f32b8038790 2024-03-18T20:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.763+0000 7f32cd7fa700 1 -- 172.21.15.98:0/111363920 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f32d00fec30 con 0x7f32b8038790 2024-03-18T20:20:01.768 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:20:01.768 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 msgr2=0x7f32b803ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 0x7f32b803ac50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006fd0 tx=0x7f32c4006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 msgr2=0x7f32d0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005660 tx=0x7f32c0005740 comp rx=0 tx=0).stop 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 shutdown_connections 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32b8038790 0x7f32b803ac50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 --2- 172.21.15.98:0/111363920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32d0105630 0x7f32d0198000 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 >> 172.21.15.98:0/111363920 conn(0x7f32d0075110 msgr2=0x7f32d00fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 shutdown_connections 2024-03-18T20:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:01.765+0000 7f32d6433700 1 -- 172.21.15.98:0/111363920 wait complete. 2024-03-18T20:20:02.131 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2024-03-18T20:20:02.131 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:20:02.131 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2024-03-18T20:20:02.152 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:20:02.152 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:02.212 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2024-03-18T20:20:02.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch host add smithi169 2024-03-18T20:20:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:02.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:02.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:02 smithi098 ceph-mon[43462]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:20:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 -- 172.21.15.98:0/1751762367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 msgr2=0x7fd360106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 --2- 172.21.15.98:0/1751762367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360106aa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd354009910 tx=0x7fd354009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 -- 172.21.15.98:0/1751762367 shutdown_connections 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 --2- 172.21.15.98:0/1751762367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360106aa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 -- 172.21.15.98:0/1751762367 >> 172.21.15.98:0/1751762367 conn(0x7fd3600fc1b0 msgr2=0x7fd3600fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 -- 172.21.15.98:0/1751762367 shutdown_connections 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.424+0000 7fd36568e700 1 -- 172.21.15.98:0/1751762367 wait complete. 2024-03-18T20:20:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd36568e700 1 Processor -- start 2024-03-18T20:20:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd36568e700 1 -- start start 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd36568e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd36568e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3601963d0 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd35effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd35effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57760/0 (socket says 172.21.15.98:57760) 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.425+0000 7fd35effd700 1 -- 172.21.15.98:0/4173029788 learned_addr learned my addr 172.21.15.98:0/4173029788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd35effd700 1 -- 172.21.15.98:0/4173029788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3540095f0 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd35effd700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd354005660 tx=0x7fd354005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd35401d070 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd360196630 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd35400b6f0 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd3540037f0 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.426+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd360196ab0 con 0x7fd3601066c0 2024-03-18T20:20:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fd354003ab0 con 0x7fd3601066c0 2024-03-18T20:20:03.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd347fff700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 0x7fd34803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:03.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd360101e00 con 0x7fd3601066c0 2024-03-18T20:20:03.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd35404c400 con 0x7fd3601066c0 2024-03-18T20:20:03.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd35e7fc700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 0x7fd34803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:03.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.427+0000 7fd35e7fc700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 0x7fd34803ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd350006fd0 tx=0x7fd350006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:03.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.432+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd35402ad80 con 0x7fd3601066c0 2024-03-18T20:20:03.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:03.627+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}) v1 -- 0x7fd360108a80 con 0x7fd3480387a0 2024-03-18T20:20:04.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:03 smithi098 ceph-mon[43462]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: Deploying cephadm binary to smithi169 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: Updating smithi136:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:05.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:04 smithi098 ceph-mon[43462]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:05.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.736+0000 7fd347fff700 1 -- 172.21.15.98:0/4173029788 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd360108a80 con 0x7fd3480387a0 2024-03-18T20:20:05.741 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi169' with addr '172.21.15.169' 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 msgr2=0x7fd34803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 0x7fd34803ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd350006fd0 tx=0x7fd350006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 msgr2=0x7fd360195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd354005660 tx=0x7fd354005740 comp rx=0 tx=0).stop 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 shutdown_connections 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd3480387a0 0x7fd34803ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 --2- 172.21.15.98:0/4173029788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3601066c0 0x7fd360195e90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 >> 172.21.15.98:0/4173029788 conn(0x7fd3600fc1b0 msgr2=0x7fd360108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 shutdown_connections 2024-03-18T20:20:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:05.738+0000 7fd36568e700 1 -- 172.21.15.98:0/4173029788 wait complete. 2024-03-18T20:20:06.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T20:20:06.284 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:06.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: Updating smithi136:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:20:06.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:06.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:06.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:06.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:06.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:06.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: Added host smithi169 2024-03-18T20:20:06.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:06.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:06.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:07.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 -- 172.21.15.98:0/2038001112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 msgr2=0x7f4944105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:07.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 --2- 172.21.15.98:0/2038001112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944105a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4934009910 tx=0x7f4934009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:07.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 -- 172.21.15.98:0/2038001112 shutdown_connections 2024-03-18T20:20:07.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 --2- 172.21.15.98:0/2038001112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944105a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:07.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 -- 172.21.15.98:0/2038001112 >> 172.21.15.98:0/2038001112 conn(0x7f49440fb110 msgr2=0x7f49440fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 -- 172.21.15.98:0/2038001112 shutdown_connections 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.255+0000 7f494949f700 1 -- 172.21.15.98:0/2038001112 wait complete. 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f494949f700 1 Processor -- start 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f494949f700 1 -- start start 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f494949f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f494949f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4944195250 con 0x7f4944105650 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f4942ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57786/0 (socket says 172.21.15.98:57786) 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.256+0000 7f4942ffd700 1 -- 172.21.15.98:0/1288017823 learned_addr learned my addr 172.21.15.98:0/1288017823 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f4942ffd700 1 -- 172.21.15.98:0/1288017823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49340095f0 con 0x7f4944105650 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f4942ffd700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4934005950 tx=0x7f4934005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f493401d070 con 0x7f4944105650 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4934009e60 con 0x7f4944105650 2024-03-18T20:20:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49441954b0 con 0x7f4944105650 2024-03-18T20:20:07.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4934022ba0 con 0x7f4944105650 2024-03-18T20:20:07.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.257+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49441959b0 con 0x7f4944105650 2024-03-18T20:20:07.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f493400f460 con 0x7f4944105650 2024-03-18T20:20:07.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f492bfff700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 0x7f492c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4944100d00 con 0x7f4944105650 2024-03-18T20:20:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f493404c700 con 0x7f4944105650 2024-03-18T20:20:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f49427fc700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 0x7f492c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.258+0000 7f49427fc700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 0x7f492c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4938006fd0 tx=0x7f4938006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:07.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.263+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4934020960 con 0x7f4944105650 2024-03-18T20:20:07.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.460+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49441959b0 con 0x7f492c0387f0 2024-03-18T20:20:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.461+0000 7f492bfff700 1 -- 172.21.15.98:0/1288017823 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f49441959b0 con 0x7f492c0387f0 2024-03-18T20:20:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:20:07.466 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 msgr2=0x7f492c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 0x7f492c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4938006fd0 tx=0x7f4938006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 msgr2=0x7f4944194d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4934005950 tx=0x7f4934005710 comp rx=0 tx=0).stop 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 shutdown_connections 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f492c0387f0 0x7f492c03acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 --2- 172.21.15.98:0/1288017823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4944105650 0x7f4944194d10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 >> 172.21.15.98:0/1288017823 conn(0x7f49440fb110 msgr2=0x7f49440fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 shutdown_connections 2024-03-18T20:20:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:07.463+0000 7f494949f700 1 -- 172.21.15.98:0/1288017823 wait complete. 2024-03-18T20:20:07.839 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T20:20:07.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T20:20:08.054 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:08.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:08.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:08 smithi098 ceph-mon[43462]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:20:08.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:08 smithi098 ceph-mon[43462]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 -- 172.21.15.98:0/1365936741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 msgr2=0x7f5b18106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 --2- 172.21.15.98:0/1365936741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18106a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009910 tx=0x7f5b08009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 -- 172.21.15.98:0/1365936741 shutdown_connections 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 --2- 172.21.15.98:0/1365936741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18106a90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 -- 172.21.15.98:0/1365936741 >> 172.21.15.98:0/1365936741 conn(0x7f5b180fa820 msgr2=0x7f5b180fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 -- 172.21.15.98:0/1365936741 shutdown_connections 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.006+0000 7f5b20040700 1 -- 172.21.15.98:0/1365936741 wait complete. 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b20040700 1 Processor -- start 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b20040700 1 -- start start 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b20040700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b20040700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b181963b0 con 0x7f5b181066b0 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b1dddc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b1dddc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57814/0 (socket says 172.21.15.98:57814) 2024-03-18T20:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b1dddc700 1 -- 172.21.15.98:0/2268553663 learned_addr learned my addr 172.21.15.98:0/2268553663 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b1dddc700 1 -- 172.21.15.98:0/2268553663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b080095f0 con 0x7f5b181066b0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b1dddc700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005660 tx=0x7f5b08005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b0801d070 con 0x7f5b181066b0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5b0800b6f0 con 0x7f5b181066b0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b18196610 con 0x7f5b181066b0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.007+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b080037f0 con 0x7f5b181066b0 2024-03-18T20:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.008+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b18196a90 con 0x7f5b181066b0 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.008+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f5b08003ab0 con 0x7f5b181066b0 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.008+0000 7f5b0effd700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 0x7f5b0403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.008+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b18101ee0 con 0x7f5b181066b0 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.008+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b0804c400 con 0x7f5b181066b0 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.009+0000 7f5b1d5db700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 0x7f5b0403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.009+0000 7f5b1d5db700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 0x7f5b0403ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b14006fd0 tx=0x7f5b14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.013+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5b08026020 con 0x7f5b181066b0 2024-03-18T20:20:09.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.209+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5b1819a760 con 0x7f5b181066b0 2024-03-18T20:20:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.262+0000 7f5b0effd700 1 -- 172.21.15.98:0/2268553663 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5b08029360 con 0x7f5b181066b0 2024-03-18T20:20:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 msgr2=0x7f5b0403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 0x7f5b0403ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b14006fd0 tx=0x7f5b14006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 msgr2=0x7f5b18195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005660 tx=0x7f5b08005740 comp rx=0 tx=0).stop 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 shutdown_connections 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b040387a0 0x7f5b0403ac60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 --2- 172.21.15.98:0/2268553663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b181066b0 0x7f5b18195e70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 >> 172.21.15.98:0/2268553663 conn(0x7f5b180fa820 msgr2=0x7f5b180fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 shutdown_connections 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:09.264+0000 7f5b20040700 1 -- 172.21.15.98:0/2268553663 wait complete. 2024-03-18T20:20:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-03-18T20:20:09.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:09 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2268553663' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T20:20:09.622 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-03-18T20:20:09.623 INFO:tasks.cephadm:Adding mon.b on smithi136 2024-03-18T20:20:09.623 INFO:tasks.cephadm:Adding mon.c on smithi169 2024-03-18T20:20:09.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c' 2024-03-18T20:20:09.881 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T20:20:09.950 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-18T20:20:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2268553663' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T20:20:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-18T20:20:10.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:10 smithi098 ceph-mon[43462]: Updating smithi169:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:10.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.927+0000 7f73a82ea700 1 -- 172.21.15.169:0/3648044655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 msgr2=0x7f73a0106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:10.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.927+0000 7f73a82ea700 1 --2- 172.21.15.169:0/3648044655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0106aa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7390009910 tx=0x7f7390009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:10.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 -- 172.21.15.169:0/3648044655 shutdown_connections 2024-03-18T20:20:10.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 --2- 172.21.15.169:0/3648044655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0106aa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:10.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 -- 172.21.15.169:0/3648044655 >> 172.21.15.169:0/3648044655 conn(0x7f73a00fc1d0 msgr2=0x7f73a00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:10.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 -- 172.21.15.169:0/3648044655 shutdown_connections 2024-03-18T20:20:10.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 -- 172.21.15.169:0/3648044655 wait complete. 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 Processor -- start 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.928+0000 7f73a82ea700 1 -- start start 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a82ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a82ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0196430 con 0x7f73a01066c0 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a6086700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a6086700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.169:60704/0 (socket says 172.21.15.169:60704) 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a6086700 1 -- 172.21.15.169:0/1297029733 learned_addr learned my addr 172.21.15.169:0/1297029733 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:20:10.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a6086700 1 -- 172.21.15.169:0/1297029733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900095f0 con 0x7f73a01066c0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73a6086700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f73900098e0 tx=0x7f7390005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.929+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f739001d070 con 0x7f73a01066c0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7390009e60 con 0x7f73a01066c0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7390022b30 con 0x7f73a01066c0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a0196690 con 0x7f73a01066c0 2024-03-18T20:20:10.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0196b10 con 0x7f73a01066c0 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f739000b960 con 0x7f73a01066c0 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.930+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a01045b0 con 0x7f73a01066c0 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.931+0000 7f73977fe700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 0x7f738c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.931+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f739004cea0 con 0x7f73a01066c0 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.931+0000 7f73a5885700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 0x7f738c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:10.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.931+0000 7f73a5885700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 0x7f738c03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f739c006fd0 tx=0x7f739c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:10.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:10.935+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f739002bba0 con 0x7f73a01066c0 2024-03-18T20:20:11.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.137+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f73a0108b60 con 0x7f738c0387a0 2024-03-18T20:20:11.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.144+0000 7f73977fe700 1 -- 172.21.15.169:0/1297029733 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f73a0108b60 con 0x7f738c0387a0 2024-03-18T20:20:11.148 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2024-03-18T20:20:11.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 msgr2=0x7f738c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:11.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 0x7f738c03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f739c006fd0 tx=0x7f739c006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 msgr2=0x7f73a0195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f73900098e0 tx=0x7f7390005710 comp rx=0 tx=0).stop 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 shutdown_connections 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f738c0387a0 0x7f738c03ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 --2- 172.21.15.169:0/1297029733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73a01066c0 0x7f73a0195ef0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.145+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 >> 172.21.15.169:0/1297029733 conn(0x7f73a00fc1d0 msgr2=0x7f73a0108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.146+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 shutdown_connections 2024-03-18T20:20:11.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:11.146+0000 7f73a82ea700 1 -- 172.21.15.169:0/1297029733 wait complete. 2024-03-18T20:20:11.578 DEBUG:teuthology.orchestra.run.smithi136:mon.b> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T20:20:11.581 DEBUG:teuthology.orchestra.run.smithi169:mon.c> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T20:20:11.583 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T20:20:11.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T20:20:11.615 INFO:journalctl@ceph.mon.c.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:20:11.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: Updating smithi169:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:11.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:12.003 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:12 smithi098 ceph-mon[43462]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:12 smithi098 ceph-mon[43462]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c;count:3 2024-03-18T20:20:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:12 smithi098 ceph-mon[43462]: Deploying daemon mon.c on smithi169 2024-03-18T20:20:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:12 smithi098 ceph-mon[43462]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:13.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 -- 172.21.15.169:0/3268605753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 msgr2=0x7f432c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 --2- 172.21.15.169:0/3268605753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c101c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f431c009910 tx=0x7f431c009c20 comp rx=0 tx=0).stop 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 -- 172.21.15.169:0/3268605753 shutdown_connections 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 --2- 172.21.15.169:0/3268605753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c101c70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 -- 172.21.15.169:0/3268605753 >> 172.21.15.169:0/3268605753 conn(0x7f432c0fb0e0 msgr2=0x7f432c0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 -- 172.21.15.169:0/3268605753 shutdown_connections 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.387+0000 7f43314ab700 1 -- 172.21.15.169:0/3268605753 wait complete. 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.388+0000 7f43314ab700 1 Processor -- start 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.388+0000 7f43314ab700 1 -- start start 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.388+0000 7f43314ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.388+0000 7f43314ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1964d0 con 0x7f432c101890 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f432affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f432affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.169:58012/0 (socket says 172.21.15.169:58012) 2024-03-18T20:20:13.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f432affd700 1 -- 172.21.15.169:0/3892197411 learned_addr learned my addr 172.21.15.169:0/3892197411 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:20:13.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f432affd700 1 -- 172.21.15.169:0/3892197411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0095f0 con 0x7f432c101890 2024-03-18T20:20:13.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f432affd700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f431c0055f0 tx=0x7f431c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.389+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c01d070 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.390+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c1966d0 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.390+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f431c00b750 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.390+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c0037b0 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.390+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c196af0 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.390+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f431c003a70 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.391+0000 7f4313fff700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 0x7f431403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.391+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f431c04c760 con 0x7f432c101890 2024-03-18T20:20:13.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.391+0000 7f432a7fc700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 0x7f431403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:13.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.391+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c069060 con 0x7f432c101890 2024-03-18T20:20:13.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.391+0000 7f432a7fc700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 0x7f431403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4320006fd0 tx=0x7f4320006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:13.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.396+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f431c026070 con 0x7f432c101890 2024-03-18T20:20:13.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.699+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f432c1972a0 con 0x7f432c101890 2024-03-18T20:20:13.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.699+0000 7f4313fff700 1 -- 172.21.15.169:0/3892197411 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f431c029770 con 0x7f432c101890 2024-03-18T20:20:13.704 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:20:13.704 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","modified":"2024-03-18T20:18:54.138830Z","created":"2024-03-18T20:18:54.138830Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.701+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 msgr2=0x7f431403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.701+0000 7f43314ab700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 0x7f431403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4320006fd0 tx=0x7f4320006e40 comp rx=0 tx=0).stop 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.701+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 msgr2=0x7f432c195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.701+0000 7f43314ab700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f431c0055f0 tx=0x7f431c0056d0 comp rx=0 tx=0).stop 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 shutdown_connections 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43140387a0 0x7f431403ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:13.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 --2- 172.21.15.169:0/3892197411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f432c101890 0x7f432c195f90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:13.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 >> 172.21.15.169:0/3892197411 conn(0x7f432c0fb0e0 msgr2=0x7f432c104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:13.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 shutdown_connections 2024-03-18T20:20:13.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:13.702+0000 7f43314ab700 1 -- 172.21.15.169:0/3892197411 wait complete. 2024-03-18T20:20:13.707 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-03-18T20:20:14.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 systemd[1]: Started Ceph mon.c for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:20:14.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T20:20:14.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.num_levels: 7 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T20:20:14.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T20:20:14.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T20:20:14.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T20:20:14.639 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.table_properties_collectors: 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.inplace_update_support: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.bloom_locality: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.max_successive_merges: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T20:20:14.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.ttl: 2592000 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.enable_blob_files: false 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.min_blob_size: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T20:20:14.641 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8cede4b2-8eeb-4713-868e-0c3f7a0e7795 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793214385106, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793214385546, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710793214, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8cede4b2-8eeb-4713-868e-0c3f7a0e7795", "db_session_id": "OJ5NR6E4L1TCVC6V2BY4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710793214385627, "job": 1, "event": "recovery_finished"} 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564d569d4000 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: DB pointer 0x564d569c2000 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T20:20:14.642 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** DB Stats ** 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T20:20:14.643 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T20:20:14.644 INFO:journalctl@ceph.mon.c.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache BinnedLRUCache@0x564d569371f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: using public_addr v2:172.21.15.169:0/0 -> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: starting mon.c rank -1 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(???) e0 preinit fsid a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).mds e1 new map 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-18T20:20:14.645 INFO:journalctl@ceph.mon.c.smithi169.stdout: e1 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout: legacy client fscid: -1 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout: 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout: No filesystems configured 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:14.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mkfs a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: fsmap 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e1: no daemons active 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/445451624' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3713959363' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3713959363' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: fsmap 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T20:20:14.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e1: no daemons active 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2443025996' entity='client.admin' 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1629574689' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2533342490' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Activating manager daemon x 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e2: x(active, starting, since 0.00302998s) 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Manager daemon x is now available 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:20:14.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14100 172.21.15.98:0/1503070806' entity='mgr.x' 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2759934608' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e3: x(active, since 1.00713s) 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e4: x(active, since 2s) 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3066743419' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/864108632' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2523259312' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2523259312' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e5: x(active, since 9s) 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3476140091' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Active manager daemon x restarted 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Activating manager daemon x 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T20:20:14.649 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e6: x(active, starting, since 0.00321997s) 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Manager daemon x is now available 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Found migration_current of "None". Setting to last migration. 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:20:14.650 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e7: x(active, since 1.00508s) 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:26] ENGINE Bus STARTING 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:26] ENGINE Serving on http://172.21.15.98:8765 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:27] ENGINE Serving on https://172.21.15.98:7150 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:27] ENGINE Bus STARTED 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e8: x(active, since 2s) 2024-03-18T20:20:14.651 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Generating ssh key... 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Deploying cephadm binary to smithi098 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Added host smithi098 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Saving service mon spec with placement count:5 2024-03-18T20:20:14.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Saving service mgr spec with placement count:2 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3603459192' entity='client.admin' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14120 172.21.15.98:0/1776689855' entity='mgr.x' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2130347759' entity='client.admin' 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1422491632' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1422491632' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e9: x(active, since 13s) 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2726802899' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Active manager daemon x restarted 2024-03-18T20:20:14.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Activating manager daemon x 2024-03-18T20:20:14.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T20:20:14.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e10: x(active, starting, since 0.00332258s) 2024-03-18T20:20:14.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:14.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T20:20:14.654 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T20:20:14.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T20:20:14.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T20:20:14.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Manager daemon x is now available 2024-03-18T20:20:14.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:45] ENGINE Bus STARTING 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:45] ENGINE Serving on https://172.21.15.98:7150 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e11: x(active, since 1.00804s) 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:45] ENGINE Serving on http://172.21.15.98:8765 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: [18/Mar/2024:20:19:45] ENGINE Bus STARTED 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.656 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e12: x(active, since 2s) 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/913871302' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/754781846' entity='client.admin' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mgrmap e13: x(active, since 6s) 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3995447124' entity='client.admin' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:14.657 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi098:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi098:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.658 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Deploying cephadm binary to smithi136 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Added host smithi136 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Deploying cephadm binary to smithi169 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi136:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi136:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Added host smithi169 2024-03-18T20:20:14.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2268553663' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2268553663' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:14.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi169:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Updating smithi169:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:14.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:14.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c;count:3 2024-03-18T20:20:14.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: Deploying daemon mon.c on smithi169 2024-03-18T20:20:14.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:14.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:14 smithi169 ceph-mon[48529]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T20:20:15.110 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T20:20:15.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T20:20:15.331 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 -- 172.21.15.169:0/924308078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280035c0 msgr2=0x7fd028005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 --2- 172.21.15.169:0/924308078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280035c0 0x7fd028005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd038068ce0 tx=0x7fd02c013e90 comp rx=0 tx=0).stop 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 -- 172.21.15.169:0/924308078 shutdown_connections 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 --2- 172.21.15.169:0/924308078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280035c0 0x7fd028005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 -- 172.21.15.169:0/924308078 >> 172.21.15.169:0/924308078 conn(0x7fd0380fb1c0 msgr2=0x7fd0380fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 -- 172.21.15.169:0/924308078 shutdown_connections 2024-03-18T20:20:19.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.550+0000 7fd03ee29700 1 -- 172.21.15.169:0/924308078 wait complete. 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 Processor -- start 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 -- start start 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 0x7fd03819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038196af0 con 0x7fd038196560 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.551+0000 7fd03ee29700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038196c60 con 0x7fd0280035c0 2024-03-18T20:20:19.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54854/0 (socket says 172.21.15.169:54854) 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 -- 172.21.15.169:0/1231477214 learned_addr learned my addr 172.21.15.169:0/1231477214 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd037fff700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 0x7fd03819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 -- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 msgr2=0x7fd038196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 -- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 msgr2=0x7fd038196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd03cbc5700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd037fff700 1 -- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 msgr2=0x7fd038196020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd037fff700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd037fff700 1 -- 172.21.15.169:0/1231477214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0138e0 con 0x7fd038196560 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.552+0000 7fd037fff700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 0x7fd03819a910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b4d0 tx=0x7fd02400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.553+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd024011630 con 0x7fd038196560 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.553+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd024011c20 con 0x7fd038196560 2024-03-18T20:20:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.553+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03819aeb0 con 0x7fd038196560 2024-03-18T20:20:19.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.553+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd024010c60 con 0x7fd038196560 2024-03-18T20:20:19.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.553+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03819b400 con 0x7fd038196560 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.554+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fd02401a400 con 0x7fd038196560 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.554+0000 7fd035ffb700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 0x7fd01c03aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.554+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd02404adc0 con 0x7fd038196560 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.554+0000 7fd03cbc5700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 0x7fd01c03aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.554+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd038105f90 con 0x7fd038196560 2024-03-18T20:20:19.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.555+0000 7fd03cbc5700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 0x7fd01c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00f470 tx=0x7fd02c00f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:19.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.561+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd024010dc0 con 0x7fd038196560 2024-03-18T20:20:19.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.834+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0380623e0 con 0x7fd038196560 2024-03-18T20:20:19.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.834+0000 7fd035ffb700 1 -- 172.21.15.169:0/1231477214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fd0240172c0 con 0x7fd038196560 2024-03-18T20:20:19.840 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:20:19.840 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","modified":"2024-03-18T20:20:14.413195Z","created":"2024-03-18T20:18:54.138830Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 msgr2=0x7fd01c03aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 0x7fd01c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00f470 tx=0x7fd02c00f3c0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 msgr2=0x7fd03819a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 0x7fd03819a910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b4d0 tx=0x7fd02400b890 comp rx=0 tx=0).stop 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 shutdown_connections 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd01c0385e0 0x7fd01c03aaa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0280035c0 0x7fd038196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 --2- 172.21.15.169:0/1231477214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd038196560 0x7fd03819a910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:19.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.836+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 >> 172.21.15.169:0/1231477214 conn(0x7fd0380fb1c0 msgr2=0x7fd03818eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:19.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.837+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 shutdown_connections 2024-03-18T20:20:19.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:19.837+0000 7fd03ee29700 1 -- 172.21.15.169:0/1231477214 wait complete. 2024-03-18T20:20:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: mon.a calling monitor election 2024-03-18T20:20:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: mon.c calling monitor election 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: fsmap 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: mgrmap e13: x(active, since 33s) 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: overall HEALTH_OK 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:19.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:19.845 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: mon.a calling monitor election 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:19.869 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: mon.c calling monitor election 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: fsmap 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: mgrmap e13: x(active, since 33s) 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: overall HEALTH_OK 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:19.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: Deploying daemon mon.b on smithi136 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/1231477214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T20:20:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: Deploying daemon mon.b on smithi136 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/1231477214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T20:20:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:21.223 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T20:20:21.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T20:20:21.453 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:20:22.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 systemd[1]: Started Ceph mon.b for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).mds e1 new map 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: e1 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: legacy client fscid: -1 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout: No filesystems configured 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T20:20:22.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.a calling monitor election 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.c calling monitor election 2024-03-18T20:20:22.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: fsmap 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mgrmap e13: x(active, since 33s) 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: overall HEALTH_OK 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:22.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: Deploying daemon mon.b on smithi136 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/1231477214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:22.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:22 smithi136 ceph-mon[48073]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T20:20:27.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:27.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: mon.a calling monitor election 2024-03-18T20:20:27.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: mon.c calling monitor election 2024-03-18T20:20:27.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:27.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: fsmap 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: mgrmap e13: x(active, since 41s) 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.595 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: mon.a calling monitor election 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: mon.c calling monitor election 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:27.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: fsmap 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: mgrmap e13: x(active, since 41s) 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-03-18T20:20:27.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:27.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.424+0000 7f0cd222f700 1 -- 172.21.15.169:0/1520116631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb8006230 msgr2=0x7f0cb80086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:28.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.424+0000 7f0cd222f700 1 --2- 172.21.15.169:0/1520116631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb8006230 0x7f0cb80086a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc068ee0 tx=0x7f0cbc00fe90 comp rx=0 tx=0).stop 2024-03-18T20:20:28.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.424+0000 7f0cd222f700 1 -- 172.21.15.169:0/1520116631 shutdown_connections 2024-03-18T20:20:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.424+0000 7f0cd222f700 1 --2- 172.21.15.169:0/1520116631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb8006230 0x7f0cb80086a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.424+0000 7f0cd222f700 1 -- 172.21.15.169:0/1520116631 >> 172.21.15.169:0/1520116631 conn(0x7f0ccc0fb260 msgr2=0x7f0ccc0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.425+0000 7f0cd222f700 1 -- 172.21.15.169:0/1520116631 shutdown_connections 2024-03-18T20:20:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.425+0000 7f0cd222f700 1 -- 172.21.15.169:0/1520116631 wait complete. 2024-03-18T20:20:28.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.425+0000 7f0cd222f700 1 Processor -- start 2024-03-18T20:20:28.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 -- start start 2024-03-18T20:20:28.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:28.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 0x7f0ccc0ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:28.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 0x7f0ccc1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc1a4d30 con 0x7f0ccc1a2370 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc1a4e70 con 0x7f0ccc0ffb70 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0cd222f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc1a5030 con 0x7f0cb8006230 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41138/0 (socket says 172.21.15.169:41138) 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 -- 172.21.15.169:0/2357498054 learned_addr learned my addr 172.21.15.169:0/2357498054 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 0x7f0ccc1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 msgr2=0x7f0ccc0ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 msgr2=0x7f0ccc0ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccb7fe700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccaffd700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 0x7f0ccc0ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:28.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 msgr2=0x7f0ccc0ff630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 msgr2=0x7f0ccc0ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 0x7f0ccc0ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccbfff700 1 -- 172.21.15.169:0/2357498054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc00f8e0 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.426+0000 7f0ccaffd700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 0x7f0ccc0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0ccbfff700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 0x7f0ccc1a4760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400bd50 tx=0x7f0cc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400cd10 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0cc400ce70 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc1a5310 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4012760 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.427+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc1a5810 con 0x7f0ccc1a2370 2024-03-18T20:20:28.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f0cc40128e0 con 0x7f0ccc1a2370 2024-03-18T20:20:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc069560 con 0x7f0ccc1a2370 2024-03-18T20:20:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0cc8ff9700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 0x7f0cac03b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0cc40146d0 con 0x7f0ccc1a2370 2024-03-18T20:20:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0ccb7fe700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 0x7f0cac03b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.428+0000 7f0ccb7fe700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 0x7f0cac03b130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc005b40 tx=0x7f0cbc005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.433+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0cc401d3b0 con 0x7f0ccc1a2370 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:28.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.719+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ccc1a5a90 con 0x7f0ccc1a2370 2024-03-18T20:20:28.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.721+0000 7f0cc8ff9700 1 -- 172.21.15.169:0/2357498054 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f0cc40112a0 con 0x7f0ccc1a2370 2024-03-18T20:20:28.726 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-18T20:20:28.726 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","modified":"2024-03-18T20:20:22.199687Z","created":"2024-03-18T20:18:54.138830Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T20:20:28.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 msgr2=0x7f0cac03b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:28.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 0x7f0cac03b130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc005b40 tx=0x7f0cbc005a90 comp rx=0 tx=0).stop 2024-03-18T20:20:28.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 msgr2=0x7f0ccc1a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 0x7f0ccc1a4760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400bd50 tx=0x7f0cc400be30 comp rx=0 tx=0).stop 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 shutdown_connections 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1a2370 0x7f0ccc1a4760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cb8006230 0x7f0ccc0ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0cac038c70 0x7f0cac03b130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 --2- 172.21.15.169:0/2357498054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ccc0ffb70 0x7f0ccc0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 >> 172.21.15.169:0/2357498054 conn(0x7f0ccc0fb260 msgr2=0x7f0ccc102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 shutdown_connections 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:28.724+0000 7f0cd222f700 1 -- 172.21.15.169:0/2357498054 wait complete. 2024-03-18T20:20:28.729 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 3 2024-03-18T20:20:29.090 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T20:20:29.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T20:20:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:29.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:20:29.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.a calling monitor election 2024-03-18T20:20:29.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.c calling monitor election 2024-03-18T20:20:29.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:20:29.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.493 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: fsmap 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mgrmap e13: x(active, since 41s) 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-03-18T20:20:29.494 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:29.495 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:29.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mon.b calling monitor election 2024-03-18T20:20:29.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mon.b calling monitor election 2024-03-18T20:20:29.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mon.a calling monitor election 2024-03-18T20:20:29.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mon.c calling monitor election 2024-03-18T20:20:29.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: fsmap 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: mgrmap e13: x(active, since 43s) 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: Cluster is now healthy 2024-03-18T20:20:29.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:29 smithi098 ceph-mon[43462]: overall HEALTH_OK 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mon.b calling monitor election 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mon.b calling monitor election 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mon.a calling monitor election 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mon.c calling monitor election 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: fsmap 2024-03-18T20:20:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:30.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: mgrmap e13: x(active, since 43s) 2024-03-18T20:20:30.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T20:20:30.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: Cluster is now healthy 2024-03-18T20:20:30.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:29 smithi169 ceph-mon[48529]: overall HEALTH_OK 2024-03-18T20:20:30.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.b calling monitor election 2024-03-18T20:20:30.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.b calling monitor election 2024-03-18T20:20:30.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.a calling monitor election 2024-03-18T20:20:30.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.c calling monitor election 2024-03-18T20:20:30.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: fsmap 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: mgrmap e13: x(active, since 43s) 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: Cluster is now healthy 2024-03-18T20:20:30.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:29 smithi136 ceph-mon[48073]: overall HEALTH_OK 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1509439910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8106df0 msgr2=0x7ff6e81071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1509439910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8106df0 0x7ff6e81071d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80099c0 tx=0x7ff6d8009cd0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1509439910 shutdown_connections 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1509439910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6e8069a10 0x7ff6e810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1509439910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8106df0 0x7ff6e81071d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1509439910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069050 0x7ff6e80694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1509439910 >> 172.21.15.98:0/1509439910 conn(0x7ff6e8076c80 msgr2=0x7ff6e8077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.595+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1509439910 shutdown_connections 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.596+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1509439910 wait complete. 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.596+0000 7ff6ed80b700 1 Processor -- start 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.596+0000 7ff6ed80b700 1 -- start start 2024-03-18T20:20:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069a10 0x7ff6e81975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6e8106df0 0x7ff6e819b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e819bf60 con 0x7ff6e8069050 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e819c0d0 con 0x7ff6e8069a10 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e819c290 con 0x7ff6e8106df0 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39496/0 (socket says 172.21.15.98:39496) 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 -- 172.21.15.98:0/1613974293 learned_addr learned my addr 172.21.15.98:0/1613974293 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e67fc700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069a10 0x7ff6e81975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 -- 172.21.15.98:0/1613974293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6e8106df0 msgr2=0x7ff6e819b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6e8106df0 0x7ff6e819b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 -- 172.21.15.98:0/1613974293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069a10 msgr2=0x7ff6e81975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069a10 0x7ff6e81975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 -- 172.21.15.98:0/1613974293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80095f0 con 0x7ff6e8069050 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6e6ffd700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005910 tx=0x7ff6d800f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d801c070 con 0x7ff6e8069050 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff6d800fe10 con 0x7ff6e8069050 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e819c510 con 0x7ff6e8069050 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8004970 con 0x7ff6e8069050 2024-03-18T20:20:30.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.597+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e81a3c00 con 0x7ff6e8069050 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.598+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e81092f0 con 0x7ff6e8069050 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.599+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7ff6d8020070 con 0x7ff6e8069050 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.599+0000 7ff6ec809700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 0x7ff6d003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.599+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff6d804c380 con 0x7ff6e8069050 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.599+0000 7ff6e67fc700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 0x7ff6d003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:30.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.599+0000 7ff6e67fc700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 0x7ff6d003ae90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6e81985c0 tx=0x7ff6dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.603+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff6d8005130 con 0x7ff6e8069050 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.793+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff6e81024a0 con 0x7ff6e8069050 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.793+0000 7ff6ec809700 1 -- 172.21.15.98:0/1613974293 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7ff6d802ab60 con 0x7ff6e8069050 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:20:30.797 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 msgr2=0x7ff6d003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 0x7ff6d003ae90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6e81985c0 tx=0x7ff6dc009260 comp rx=0 tx=0).stop 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 msgr2=0x7ff6e81970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005910 tx=0x7ff6d800f670 comp rx=0 tx=0).stop 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 shutdown_connections 2024-03-18T20:20:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6e8106df0 0x7ff6e819b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e8069050 0x7ff6e81970b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff6d00389d0 0x7ff6d003ae90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 --2- 172.21.15.98:0/1613974293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6e8069a10 0x7ff6e81975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 >> 172.21.15.98:0/1613974293 conn(0x7ff6e8076c80 msgr2=0x7ff6e80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 shutdown_connections 2024-03-18T20:20:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:30.795+0000 7ff6ed80b700 1 -- 172.21.15.98:0/1613974293 wait complete. 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:30 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1613974293' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:31.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:31.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:30 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1613974293' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.177 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T20:20:31.177 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:20:31.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T20:20:31.207 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:20:31.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:31.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:20:31.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:31.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:30 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1613974293' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:31.275 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:20:31.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T20:20:31.307 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:20:31.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:31.374 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:20:31.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T20:20:31.408 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:20:31.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T20:20:31.478 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-03-18T20:20:31.478 INFO:tasks.cephadm:Adding mgr.y on smithi136 2024-03-18T20:20:31.478 INFO:tasks.cephadm:Adding mgr.z on smithi169 2024-03-18T20:20:31.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi098=x;smithi136=y;smithi169=z' 2024-03-18T20:20:31.989 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: Reconfiguring mon.b (monmap changed)... 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: Reconfiguring daemon mon.b on smithi136 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:32.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:32.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: Reconfiguring mon.b (monmap changed)... 2024-03-18T20:20:32.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: Reconfiguring daemon mon.b on smithi136 2024-03-18T20:20:32.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:32.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:32.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: Reconfiguring mon.b (monmap changed)... 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: Reconfiguring daemon mon.b on smithi136 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T20:20:32.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:33.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/1781131242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 msgr2=0x7fdd54108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/1781131242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd54108140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00b0a0 tx=0x7fdd4c00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/1781131242 shutdown_connections 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/1781131242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd54108140 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/1781131242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 0x7fdd54100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/1781131242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 0x7fdd54101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/1781131242 >> 172.21.15.169:0/1781131242 conn(0x7fdd540fb2b0 msgr2=0x7fdd540fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/1781131242 shutdown_connections 2024-03-18T20:20:33.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/1781131242 wait complete. 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 Processor -- start 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.024+0000 7fdd5aa6b700 1 -- start start 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 0x7fdd54196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 0x7fdd54197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5419e7a0 con 0x7fdd54100ef0 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5419e8e0 con 0x7fdd540ffcf0 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5419eaa0 con 0x7fdd54101890 2024-03-18T20:20:33.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41150/0 (socket says 172.21.15.169:41150) 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 -- 172.21.15.169:0/3064451611 learned_addr learned my addr 172.21.15.169:0/3064451611 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd53fff700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 0x7fdd54197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd58807700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 0x7fdd54196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 -- 172.21.15.169:0/3064451611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 msgr2=0x7fdd54196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 0x7fdd54196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 -- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 msgr2=0x7fdd54197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:33.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 0x7fdd54197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 -- 172.21.15.169:0/3064451611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c009d00 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd59008700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c004d00 tx=0x7fdd4c004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c00e040 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd4c006060 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5419ed20 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.025+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c01b950 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.026+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5419f190 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.026+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fdd4c01bab0 con 0x7fdd54101890 2024-03-18T20:20:33.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.026+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5404ea80 con 0x7fdd54101890 2024-03-18T20:20:33.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.026+0000 7fdd51ffb700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 0x7fdd4403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:33.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.027+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd4c04cf00 con 0x7fdd54101890 2024-03-18T20:20:33.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.027+0000 7fdd58807700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 0x7fdd4403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:33.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.027+0000 7fdd58807700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 0x7fdd4403ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd54198130 tx=0x7fdd48005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:33.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.032+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd4c01a3b0 con 0x7fdd54101890 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: Reconfiguring mon.c (monmap changed)... 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: Reconfiguring daemon mon.c on smithi169 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:33.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:33.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:33.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.235+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi136=y;smithi169=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd5419f3b0 con 0x7fdd440389d0 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: Reconfiguring mon.c (monmap changed)... 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: Reconfiguring daemon mon.c on smithi169 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:33.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:33.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.241+0000 7fdd51ffb700 1 -- 172.21.15.169:0/3064451611 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdd5419f3b0 con 0x7fdd440389d0 2024-03-18T20:20:33.246 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mgr update... 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 msgr2=0x7fdd4403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 0x7fdd4403ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd54198130 tx=0x7fdd48005d90 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 msgr2=0x7fdd5419e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c004d00 tx=0x7fdd4c004de0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 shutdown_connections 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd54101890 0x7fdd5419e1d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd440389d0 0x7fdd4403ae90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd540ffcf0 0x7fdd54196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 --2- 172.21.15.169:0/3064451611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd54100ef0 0x7fdd54197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 >> 172.21.15.169:0/3064451611 conn(0x7fdd540fb2b0 msgr2=0x7fdd540fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 shutdown_connections 2024-03-18T20:20:33.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:20:33.243+0000 7fdd5aa6b700 1 -- 172.21.15.169:0/3064451611 wait complete. 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: Reconfiguring mon.c (monmap changed)... 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: Reconfiguring daemon mon.c on smithi169 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:33.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:33.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:33.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:33.656 DEBUG:teuthology.orchestra.run.smithi136:mgr.y> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T20:20:33.659 DEBUG:teuthology.orchestra.run.smithi169:mgr.z> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T20:20:33.662 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T20:20:33.662 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:20:33.662 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T20:20:33.684 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T20:20:33.684 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-18T20:20:33.690 INFO:journalctl@ceph.mgr.z.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:20:33.691 INFO:journalctl@ceph.mgr.y.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 178733 Links: 1 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-18 20:19:54.552763774 +0000 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-18 20:13:34.392862184 +0000 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-18 20:13:34.392862184 +0000 2024-03-18T20:20:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-03-18T20:20:33.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T20:20:33.815 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-18T20:20:33.816 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-18T20:20:33.816 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000282044 s, 1.8 MB/s 2024-03-18T20:20:33.817 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T20:20:33.878 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 177051 Links: 1 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-18 20:19:54.605763103 +0000 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-18 20:13:34.724858252 +0000 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-18 20:13:34.724858252 +0000 2024-03-18T20:20:33.938 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-03-18T20:20:33.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T20:20:34.006 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-18T20:20:34.007 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-18T20:20:34.007 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000225138 s, 2.3 MB/s 2024-03-18T20:20:34.008 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T20:20:34.067 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-18T20:20:34.126 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T20:20:34.126 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:34.126 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 180546 Links: 1 2024-03-18T20:20:34.126 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:34.127 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:34.127 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-18 20:19:54.654762482 +0000 2024-03-18T20:20:34.127 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-18 20:13:35.084853989 +0000 2024-03-18T20:20:34.127 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-18 20:13:35.084853989 +0000 2024-03-18T20:20:34.127 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-03-18T20:20:34.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T20:20:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-18T20:20:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-18T20:20:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000194383 s, 2.6 MB/s 2024-03-18T20:20:34.195 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T20:20:34.255 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 182520 Links: 1 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-18 20:19:54.709761785 +0000 2024-03-18T20:20:34.315 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-18 20:13:35.433849855 +0000 2024-03-18T20:20:34.316 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-18 20:13:35.433849855 +0000 2024-03-18T20:20:34.316 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-03-18T20:20:34.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T20:20:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-18T20:20:34.383 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-18T20:20:34.383 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000239193 s, 2.1 MB/s 2024-03-18T20:20:34.384 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T20:20:34.442 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-18T20:20:34.442 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T20:20:34.464 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T20:20:34.464 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-18T20:20:34.524 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 179800 Links: 1 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-18 20:20:21.756178723 +0000 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-18 20:13:35.568437767 +0000 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-18 20:13:35.568437767 +0000 2024-03-18T20:20:34.525 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-18T20:20:34.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T20:20:34.586 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi136=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: Saving service mgr spec with placement smithi098=x;smithi136=y;smithi169=z;count:3 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: Deploying daemon mgr.z on smithi169 2024-03-18T20:20:34.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:34 smithi169 ceph-mon[48529]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi136=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: Saving service mgr spec with placement smithi098=x;smithi136=y;smithi169=z;count:3 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:34.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: Deploying daemon mgr.z on smithi169 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:34 smithi098 ceph-mon[43462]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi136=y;smithi169=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: Saving service mgr spec with placement smithi098=x;smithi136=y;smithi169=z;count:3 2024-03-18T20:20:34.594 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: Deploying daemon mgr.z on smithi169 2024-03-18T20:20:34.595 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:34 smithi136 ceph-mon[48073]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:34.596 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-18T20:20:34.596 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-18T20:20:34.596 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00021133 s, 2.4 MB/s 2024-03-18T20:20:34.597 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T20:20:34.659 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 179875 Links: 1 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:34.719 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-18 20:20:21.756178723 +0000 2024-03-18T20:20:34.720 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-18 20:13:35.928429921 +0000 2024-03-18T20:20:34.720 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-18 20:13:35.928429921 +0000 2024-03-18T20:20:34.720 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-18T20:20:34.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T20:20:34.787 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-18T20:20:34.787 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-18T20:20:34.788 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000238178 s, 2.1 MB/s 2024-03-18T20:20:34.789 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T20:20:34.849 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 182446 Links: 1 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-18 20:20:21.756178723 +0000 2024-03-18T20:20:34.908 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-18 20:13:36.317421442 +0000 2024-03-18T20:20:34.909 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-18 20:13:36.317421442 +0000 2024-03-18T20:20:34.909 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-18T20:20:34.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T20:20:34.976 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-18T20:20:34.976 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-18T20:20:34.976 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000225028 s, 2.3 MB/s 2024-03-18T20:20:34.977 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T20:20:35.037 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 182507 Links: 1 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-18 20:20:21.756178723 +0000 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-18 20:13:36.669413770 +0000 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-18 20:13:36.669413770 +0000 2024-03-18T20:20:35.096 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-03-18T20:20:35.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T20:20:35.164 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-18T20:20:35.164 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-18T20:20:35.164 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000263251 s, 1.9 MB/s 2024-03-18T20:20:35.165 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T20:20:35.222 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-18T20:20:35.222 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T20:20:35.241 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T20:20:35.242 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 177875 Links: 1 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-18 20:20:34.531673607 +0000 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-18 20:13:38.150268993 +0000 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-18 20:13:38.150268993 +0000 2024-03-18T20:20:35.300 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-03-18T20:20:35.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T20:20:35.366 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 bash[50372]: aaaa03a83ba203408988c4b27dcc9423295f6e5601ff3fc236e3503bc58ed364 2024-03-18T20:20:35.367 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 systemd[1]: Started Ceph mgr.z for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:20:35.370 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-18T20:20:35.370 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-18T20:20:35.370 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000241675 s, 2.1 MB/s 2024-03-18T20:20:35.371 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:35.365+0000 7f5978a3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:20:35.372 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T20:20:35.430 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-18T20:20:35.488 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 178537 Links: 1 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-18 20:20:34.531673607 +0000 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-18 20:13:38.498261790 +0000 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-18 20:13:38.498261790 +0000 2024-03-18T20:20:35.489 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-03-18T20:20:35.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T20:20:35.555 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-18T20:20:35.556 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-18T20:20:35.556 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000242269 s, 2.1 MB/s 2024-03-18T20:20:35.557 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T20:20:35.615 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-18T20:20:35.634 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:35.461+0000 7f5978a3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:20:35.634 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:35.558+0000 7f5978a3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:20:35.651 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T20:20:35.651 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:35.651 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 179714 Links: 1 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-18 20:20:34.531673607 +0000 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-18 20:13:38.849254524 +0000 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-18 20:13:38.849254524 +0000 2024-03-18T20:20:35.652 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-03-18T20:20:35.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T20:20:35.717 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-18T20:20:35.717 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-18T20:20:35.718 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00019165 s, 2.7 MB/s 2024-03-18T20:20:35.719 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T20:20:35.777 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-18T20:20:35.834 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 179826 Links: 1 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-18 20:20:34.531673607 +0000 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-18 20:13:39.235246534 +0000 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-18 20:13:39.235246534 +0000 2024-03-18T20:20:35.835 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-03-18T20:20:35.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T20:20:35.901 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-18T20:20:35.901 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-18T20:20:35.901 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00022618 s, 2.3 MB/s 2024-03-18T20:20:35.902 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T20:20:35.960 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-18T20:20:35.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T20:20:36.103 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:35.650+0000 7f5978a3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:20:36.188 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: Deploying daemon mgr.y on smithi136 2024-03-18T20:20:36.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mon[48073]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: Deploying daemon mgr.y on smithi136 2024-03-18T20:20:36.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:36 smithi098 ceph-mon[43462]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:36.306+0000 7f5978a3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T20:20:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:36.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:36.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: Deploying daemon mgr.y on smithi136 2024-03-18T20:20:36.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-mon[48529]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:36.491 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:36 smithi136 systemd[1]: Starting Ceph mgr.y for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:20:36.884 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:36.559+0000 7f5978a3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:20:36.884 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:36.644+0000 7f5978a3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:20:36.884 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:36.754+0000 7f5978a3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:20:37.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:36 smithi136 ceph-mgr[49577]: -- 172.21.15.136:0/142312010 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f07c961a0 con 0x564f07c6d000 2024-03-18T20:20:37.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:36 smithi136 bash[49540]: ce4ceee50fc19f556d6585a006620f753f059e75c6968dd5f2c5ad007ad90ff2 2024-03-18T20:20:37.242 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:36 smithi136 systemd[1]: Started Ceph mgr.y for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:20:37.376 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:37.265+0000 7fb54a52a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:20:37.384 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:36.924+0000 7f5978a3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:20:37.384 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:37.053+0000 7f5978a3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:20:37.384 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:37.145+0000 7f5978a3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112316 s, 934 MB/s 2024-03-18T20:20:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-18T20:20:37.564 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:37.370+0000 7fb54a52a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:20:37.564 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:37.475+0000 7fb54a52a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:20:37.564 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:37.558+0000 7fb54a52a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:20:37.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-18T20:20:37.884 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:37.566+0000 7f5978a3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:20:38.056 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.239 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.274 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:37.979+0000 7f5978a3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:20:38.274 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:38.175+0000 7f5978a3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:20:38.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:38.491 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.233+0000 7fb54a52a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:20:38.491 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.483+0000 7fb54a52a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:20:38.634 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:38.269+0000 7f5978a3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:20:38.853 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.568+0000 7fb54a52a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:20:38.853 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.677+0000 7fb54a52a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:20:38.853 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.847+0000 7fb54a52a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:20:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T20:20:39.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:39.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:39.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:39 smithi098 ceph-mon[43462]: Reconfiguring daemon mgr.x on smithi098 2024-03-18T20:20:39.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:38 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:38.975+0000 7fb54a52a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:20:39.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:39.061+0000 7fb54a52a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:20:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-mon[48073]: Reconfiguring daemon mgr.x on smithi098 2024-03-18T20:20:39.285 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:39.034+0000 7f5978a3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:20:39.285 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:39.131+0000 7f5978a3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:20:39.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-mon[48529]: Reconfiguring daemon mgr.x on smithi098 2024-03-18T20:20:39.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 -- 172.21.15.98:0/2834171549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c106ae0 msgr2=0x7f594c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 --2- 172.21.15.98:0/2834171549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c106ae0 0x7f594c108f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f594400b0a0 tx=0x7f594400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 -- 172.21.15.98:0/2834171549 shutdown_connections 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 --2- 172.21.15.98:0/2834171549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c106ae0 0x7f594c108f50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 --2- 172.21.15.98:0/2834171549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c101880 0x7f594c103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 --2- 172.21.15.98:0/2834171549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 0x7f594c1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 -- 172.21.15.98:0/2834171549 >> 172.21.15.98:0/2834171549 conn(0x7f594c0fb2b0 msgr2=0x7f594c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 -- 172.21.15.98:0/2834171549 shutdown_connections 2024-03-18T20:20:39.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.399+0000 7f59525ed700 1 -- 172.21.15.98:0/2834171549 wait complete. 2024-03-18T20:20:39.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 Processor -- start 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 -- start start 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 0x7f594c0ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 0x7f594c0ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c100510 con 0x7f594c101880 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c100680 con 0x7f594c1041b0 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.400+0000 7f59525ed700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c1a0d90 con 0x7f594c106ae0 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51908/0 (socket says 172.21.15.98:51908) 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 -- 172.21.15.98:0/1682612999 learned_addr learned my addr 172.21.15.98:0/1682612999 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594b7fe700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 0x7f594c0ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f5950b8a700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 0x7f594c0ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 -- 172.21.15.98:0/1682612999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 msgr2=0x7f594c0ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 0x7f594c0ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 -- 172.21.15.98:0/1682612999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 msgr2=0x7f594c0ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 0x7f594c0ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 -- 172.21.15.98:0/1682612999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5944009d00 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f5950b8a700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 0x7f594c0ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f594bfff700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f593c00e9a0 tx=0x7f593c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c009790 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f593c010430 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c1a0fc0 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c00cea0 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.401+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c1a1430 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.402+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f593c018680 con 0x7f594c101880 2024-03-18T20:20:39.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.402+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c0fce30 con 0x7f594c101880 2024-03-18T20:20:39.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.402+0000 7f59497fa700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 0x7f593403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:39.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.402+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f593c014070 con 0x7f594c101880 2024-03-18T20:20:39.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.402+0000 7f594b7fe700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 0x7f593403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:39.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.403+0000 7f594b7fe700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 0x7f593403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5940009c60 tx=0x7f5940009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:39.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.407+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f593c0173d0 con 0x7f594c101880 2024-03-18T20:20:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:39.608+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f594c0611f0 con 0x7f59340389d0 2024-03-18T20:20:39.634 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:39.280+0000 7f5978a3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:20:39.741 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:39.437+0000 7fb54a52a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:20:40.129 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:39 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:39.838+0000 7fb54a52a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:20:40.130 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:40.031+0000 7fb54a52a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:20:40.221 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:39 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:39.914+0000 7f5978a3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:20:40.221 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:40.098+0000 7f5978a3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:20:40.491 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:40.124+0000 7fb54a52a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:20:40.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:40.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:40.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:40.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:40.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:40.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:40.634 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:40.295+0000 7f5978a3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:20:40.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:40.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:40.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:40.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:40.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:40.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:41.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:40.884+0000 7fb54a52a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:20:41.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:40 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:40.980+0000 7fb54a52a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:20:41.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:41.127+0000 7fb54a52a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:20:41.516 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:41.225+0000 7f5978a3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:20:41.516 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:41.511+0000 7f5978a3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:41.599+0000 7f5978a3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]: dispatch 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]': finished 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-mon[48529]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T20:20:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:41.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:41.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]: dispatch 2024-03-18T20:20:41.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]': finished 2024-03-18T20:20:41.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:41 smithi098 ceph-mon[43462]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T20:20:41.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:41.938 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:41.753+0000 7fb54a52a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:20:41.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:41.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]: dispatch 2024-03-18T20:20:41.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/751189885' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d91034c-3502-4016-9090-515df19fa756"}]': finished 2024-03-18T20:20:41.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-mon[48073]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T20:20:41.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:42.134 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:20:41 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z[50387]: 2024-03-18T20:20:41.792+0000 7f5978a3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:20:42.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:41 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:41.933+0000 7fb54a52a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:20:42.241 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:42.125+0000 7fb54a52a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:20:42.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:42.586+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) v1 ==== 90699+0+0 (secure 0 0 0) 0x7f593c0105a0 con 0x7f594c101880 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: Standby manager daemon z started 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:42.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:42 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1386566025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: Standby manager daemon z started 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:42 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1386566025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: Standby manager daemon z started 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T20:20:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.169:0/245671930' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:42.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:42 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1386566025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:20:43.342 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:43.054+0000 7fb54a52a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:20:43.617 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:43.337+0000 7fb54a52a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:20:43.617 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:43.424+0000 7fb54a52a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:20:43.618 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-mon[48073]: mgrmap e14: x(active, since 57s), standbys: z 2024-03-18T20:20:43.618 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T20:20:43.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:43 smithi098 ceph-mon[43462]: mgrmap e14: x(active, since 57s), standbys: z 2024-03-18T20:20:43.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T20:20:43.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:43 smithi169 ceph-mon[48529]: mgrmap e14: x(active, since 57s), standbys: z 2024-03-18T20:20:43.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T20:20:43.991 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:20:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y[49554]: 2024-03-18T20:20:43.613+0000 7fb54a52a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:20:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:44.599+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7f593c016c40 con 0x7f594c101880 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: Standby manager daemon y started 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T20:20:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:44 smithi169 ceph-mon[48529]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: Standby manager daemon y started 2024-03-18T20:20:44.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T20:20:44.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:44.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T20:20:44.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:44 smithi136 ceph-mon[48073]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: Standby manager daemon y started 2024-03-18T20:20:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T20:20:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T20:20:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T20:20:45.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:44 smithi098 ceph-mon[43462]: from='mgr.? 172.21.15.136:0/789897949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T20:20:45.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:45 smithi169 ceph-mon[48529]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-03-18T20:20:45.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T20:20:45.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:45 smithi098 ceph-mon[43462]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-03-18T20:20:45.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T20:20:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:45 smithi136 ceph-mon[48073]: mgrmap e15: x(active, since 59s), standbys: z, y 2024-03-18T20:20:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T20:20:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:46 smithi169 ceph-mon[48529]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:46.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:46 smithi098 ceph-mon[43462]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:46 smithi136 ceph-mon[48073]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:47.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T20:20:47.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:47.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T20:20:47.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T20:20:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:48.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:48 smithi098 ceph-mon[43462]: Deploying daemon osd.0 on smithi098 2024-03-18T20:20:48.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:48 smithi098 ceph-mon[43462]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:48.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:48 smithi169 ceph-mon[48529]: Deploying daemon osd.0 on smithi098 2024-03-18T20:20:48.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:48 smithi169 ceph-mon[48529]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:48 smithi136 ceph-mon[48073]: Deploying daemon osd.0 on smithi098 2024-03-18T20:20:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:48 smithi136 ceph-mon[48073]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:50.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:50 smithi098 ceph-mon[43462]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:50.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:50.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:50 smithi169 ceph-mon[48529]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:50.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:50 smithi136 ceph-mon[48073]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:50.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:50.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.656+0000 7f59497fa700 1 -- 172.21.15.98:0/1682612999 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f594c0611f0 con 0x7f59340389d0 2024-03-18T20:20:52.660 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-03-18T20:20:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 msgr2=0x7f593403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 0x7f593403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5940009c60 tx=0x7f5940009260 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 msgr2=0x7f594c101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f593c00e9a0 tx=0x7f593c00ecb0 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 shutdown_connections 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f594c106ae0 0x7f594c0ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f594c101880 0x7f594c101190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f59340389d0 0x7f593403ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 --2- 172.21.15.98:0/1682612999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f594c1041b0 0x7f594c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 >> 172.21.15.98:0/1682612999 conn(0x7f594c0fb2b0 msgr2=0x7f594c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 shutdown_connections 2024-03-18T20:20:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:20:52.658+0000 7f59525ed700 1 -- 172.21.15.98:0/1682612999 wait complete. 2024-03-18T20:20:52.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:52.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:52.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:52.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:52.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:52.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:52.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:52.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:52.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:53.028 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T20:20:53.031 INFO:tasks.cephadm:Deploying osd.1 on smithi136 with /dev/vg_nvme/lv_4... 2024-03-18T20:20:53.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T20:20:53.062 INFO:journalctl@ceph.osd.0.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:53.064 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.065 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.065 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.065 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:20:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0093704 s, 1.1 GB/s 2024-03-18T20:20:54.591 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-18T20:20:54.816 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:20:54 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:20:54.526+0000 7f08222fe640 -1 osd.0 0 log_to_monitors true 2024-03-18T20:20:54.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:54 smithi136 ceph-mon[48073]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:54.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:54 smithi136 ceph-mon[48073]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T20:20:54.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-03-18T20:20:55.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:54 smithi098 ceph-mon[43462]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:55.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:54 smithi098 ceph-mon[43462]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T20:20:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:54 smithi169 ceph-mon[48529]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:54 smithi169 ceph-mon[48529]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T20:20:55.236 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:20:55.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T20:20:55.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:20:55.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T20:20:55.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:55 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:56.093 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:20:55.813+0000 7f0817962700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T20:20:56.093 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:20:55.815+0000 7f0811753700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:20:56.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T20:20:56.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T20:20:56.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:20:56.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T20:20:56.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:55 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:56.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:20:56.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T20:20:56.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:55 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 -- 172.21.15.136:0/1338116726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4101920 msgr2=0x7f2fc4103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1338116726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4101920 0x7f2fc4103d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009990 tx=0x7f2fb8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 -- 172.21.15.136:0/1338116726 shutdown_connections 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1338116726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4106b80 0x7f2fc4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1338116726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4101920 0x7f2fc4103d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1338116726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc4104250 0x7f2fc4106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 -- 172.21.15.136:0/1338116726 >> 172.21.15.136:0/1338116726 conn(0x7f2fc40fb2b0 msgr2=0x7f2fc40fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:20:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 -- 172.21.15.136:0/1338116726 shutdown_connections 2024-03-18T20:20:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 -- 172.21.15.136:0/1338116726 wait complete. 2024-03-18T20:20:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.249+0000 7f2fcafae700 1 Processor -- start 2024-03-18T20:20:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 -- start start 2024-03-18T20:20:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 0x7f2fc4196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc4106b80 0x7f2fc419e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc419e7f0 con 0x7f2fc4106b80 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc419e930 con 0x7f2fc4104250 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc419eaf0 con 0x7f2fc4101920 2024-03-18T20:20:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48306/0 (socket says 172.21.15.136:48306) 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 -- 172.21.15.136:0/1324948081 learned_addr learned my addr 172.21.15.136:0/1324948081 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc8d4a700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 0x7f2fc4196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 -- 172.21.15.136:0/1324948081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 msgr2=0x7f2fc4196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 0x7f2fc4196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 -- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc4106b80 msgr2=0x7f2fc419e220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc4106b80 0x7f2fc419e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 -- 172.21.15.136:0/1324948081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80095f0 con 0x7f2fc4104250 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc8d4a700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 0x7f2fc4196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:20:56.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc3fff700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000d980 tx=0x7f2fb000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000bb60 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc419edd0 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2fb0007640 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.250+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0004d60 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.251+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc419f270 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.252+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc40fce30 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.252+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7f2fb0004020 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.253+0000 7f2fc1ffb700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 0x7f2fb40a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.253+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7f2fb00cac30 con 0x7f2fc4104250 2024-03-18T20:20:56.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.253+0000 7f2fc8d4a700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 0x7f2fb40a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:20:56.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.254+0000 7f2fc8d4a700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 0x7f2fb40a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5c0 tx=0x7f2fb8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:20:56.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.258+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2fb0095f50 con 0x7f2fc4104250 2024-03-18T20:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:20:56.458+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fc40611f0 con 0x7f2fb40a1090 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi098 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi098 to 17114M 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi098 2024-03-18T20:20:57.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi098 to 17114M 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:57.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi098 2024-03-18T20:20:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi098 to 17114M 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:20:57.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:57.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:20:57.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:20:57.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:57.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] boot 2024-03-18T20:20:57.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T20:20:57.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:57 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] boot 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T20:20:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:57 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: osd.0 [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] boot 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T20:20:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:57 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/4004913884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]': finished 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T20:20:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/4004913884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]': finished 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T20:20:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:20:59.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/4004913884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]: dispatch 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf985f1e-4bad-4e71-9cf6-abea91ae0378"}]': finished 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T20:20:59.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:00.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:20:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/328491273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:00.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:20:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/328491273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:00.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:20:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/328491273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:01.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:00 smithi098 ceph-mon[43462]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:01.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:00 smithi169 ceph-mon[48529]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:01.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:00 smithi136 ceph-mon[48073]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:02 smithi169 ceph-mon[48529]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:02 smithi136 ceph-mon[48073]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:02 smithi098 ceph-mon[43462]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:04.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:04 smithi098 ceph-mon[43462]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:04.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T20:21:04.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:04.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:04 smithi169 ceph-mon[48529]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:04.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T20:21:04.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:04.888 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:04 smithi136 ceph-mon[48073]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:04.888 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T20:21:04.888 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:05.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:05 smithi136 ceph-mon[48073]: Deploying daemon osd.1 on smithi136 2024-03-18T20:21:05.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:05 smithi098 ceph-mon[43462]: Deploying daemon osd.1 on smithi136 2024-03-18T20:21:05.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:05 smithi169 ceph-mon[48529]: Deploying daemon osd.1 on smithi136 2024-03-18T20:21:06.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:06 smithi136 ceph-mon[48073]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:06.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:06 smithi098 ceph-mon[43462]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:06 smithi169 ceph-mon[48529]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:08.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:08 smithi169 ceph-mon[48529]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:08.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:08.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:08.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:08.915 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:08 smithi136 ceph-mon[48073]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:08.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:08.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:08.916 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:08 smithi098 ceph-mon[43462]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:09.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:10.315 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.315 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.315 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:10.315 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:10.316 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:10.316 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.414+0000 7f2fc1ffb700 1 -- 172.21.15.136:0/1324948081 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2fc40611f0 con 0x7f2fb40a1090 2024-03-18T20:21:10.419 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 1 on host 'smithi136' 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 msgr2=0x7f2fb40a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 0x7f2fb40a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5c0 tx=0x7f2fb8005fb0 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 msgr2=0x7f2fc4197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000d980 tx=0x7f2fb000dd40 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 shutdown_connections 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc4106b80 0x7f2fc419e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fc4101920 0x7f2fc4196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fb40a1090 0x7f2fb40a3550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 --2- 172.21.15.136:0/1324948081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fc4104250 0x7f2fc4197200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 >> 172.21.15.136:0/1324948081 conn(0x7f2fc40fb2b0 msgr2=0x7f2fc41037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 shutdown_connections 2024-03-18T20:21:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:10.416+0000 7f2fcafae700 1 -- 172.21.15.136:0/1324948081 wait complete. 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:10.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:10.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:10.856 DEBUG:teuthology.orchestra.run.smithi136:osd.1> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T20:21:10.858 INFO:tasks.cephadm:Deploying osd.2 on smithi169 with /dev/vg_nvme/lv_4... 2024-03-18T20:21:10.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T20:21:10.889 INFO:journalctl@ceph.osd.1.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:21:11.086 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:21:11.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:11 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:11 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:11 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:11.629 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:11 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.629 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:11 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.629 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:11 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:11.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:11.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:11 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:12.390 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T20:21:12.390 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T20:21:12.391 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-18T20:21:12.391 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-18T20:21:12.391 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113417 s, 925 MB/s 2024-03-18T20:21:12.391 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-18T20:21:12.401 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:21:12.026+0000 7fa5085c8640 -1 osd.1 0 log_to_monitors true 2024-03-18T20:21:12.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:12 smithi169 ceph-mon[48529]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:12.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:12 smithi169 ceph-mon[48529]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:12 smithi169 ceph-mon[48529]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:12 smithi136 ceph-mon[48073]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:12 smithi136 ceph-mon[48073]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:12 smithi136 ceph-mon[48073]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-03-18T20:21:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:12 smithi098 ceph-mon[43462]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:12 smithi098 ceph-mon[43462]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:12 smithi098 ceph-mon[43462]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T20:21:12.996 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:21:13.741 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:21:13.404+0000 7fa4fdc2c700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T20:21:13.741 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:21:13.408+0000 7fa4f7a1d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:13.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:13.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T20:21:13.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T20:21:13.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:13.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:13.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:13.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:13.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:13.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:13.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 -- 172.21.15.169:0/1362070645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 msgr2=0x7f705c108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 --2- 172.21.15.169:0/1362070645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c108140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f704c00b0a0 tx=0x7f704c00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 -- 172.21.15.169:0/1362070645 shutdown_connections 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 --2- 172.21.15.169:0/1362070645 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c108140 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 --2- 172.21.15.169:0/1362070645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0ffcf0 0x7f705c100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 --2- 172.21.15.169:0/1362070645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c100ef0 0x7f705c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 -- 172.21.15.169:0/1362070645 >> 172.21.15.169:0/1362070645 conn(0x7f705c0fb2b0 msgr2=0x7f705c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 -- 172.21.15.169:0/1362070645 shutdown_connections 2024-03-18T20:21:14.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.019+0000 7f7060ac1700 1 -- 172.21.15.169:0/1362070645 wait complete. 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 Processor -- start 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 -- start start 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c0ffcf0 0x7f705c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c100ef0 0x7f705c1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c19e8e0 con 0x7f705c100ef0 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c19ea20 con 0x7f705c0ffcf0 2024-03-18T20:21:14.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f7060ac1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c19ebb0 con 0x7f705c101890 2024-03-18T20:21:14.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f705ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:14.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f705ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58104/0 (socket says 172.21.15.169:58104) 2024-03-18T20:21:14.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.020+0000 7f705ad9d700 1 -- 172.21.15.169:0/2745309747 learned_addr learned my addr 172.21.15.169:0/2745309747 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:21:14.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705a59c700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c0ffcf0 0x7f705c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:14.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 -- 172.21.15.169:0/2745309747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c0ffcf0 msgr2=0x7f705c196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c0ffcf0 0x7f705c196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 -- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c100ef0 msgr2=0x7f705c1972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f7059d9b700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c100ef0 0x7f705c1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c100ef0 0x7f705c1972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 -- 172.21.15.169:0/2745309747 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009d00 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f705ad9d700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f704c007a60 tx=0x7f704c004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c00e040 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f704c01db20 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c19ee30 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c01bc50 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.021+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c19f240 con 0x7f705c101890 2024-03-18T20:21:14.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.022+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c107120 con 0x7f705c101890 2024-03-18T20:21:14.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.023+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7f704c01dc90 con 0x7f705c101890 2024-03-18T20:21:14.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.024+0000 7f70537fe700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 0x7f70480a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:14.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.024+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7f704c0cd280 con 0x7f705c101890 2024-03-18T20:21:14.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.024+0000 7f705a59c700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 0x7f70480a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:14.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.024+0000 7f705a59c700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 0x7f70480a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7054005fd0 tx=0x7f7054005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:14.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.027+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f704c098420 con 0x7f705c101890 2024-03-18T20:21:14.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:14.226+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f705c0611f0 con 0x7f70480a1110 2024-03-18T20:21:14.669 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi136 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi136 to 17113M 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:14.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:14.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi136 2024-03-18T20:21:14.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi136 to 17113M 2024-03-18T20:21:14.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:21:14.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:14.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:14.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi136 2024-03-18T20:21:14.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi136 to 17113M 2024-03-18T20:21:14.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:21:14.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T20:21:14.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:14.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T20:21:14.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:14.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:14.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:15.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:21:15.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:21:15.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:15.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] boot 2024-03-18T20:21:15.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T20:21:15.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] boot 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T20:21:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: osd.1 [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] boot 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T20:21:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T20:21:16.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T20:21:16.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:16.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/961955494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]': finished 2024-03-18T20:21:16.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T20:21:16.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/961955494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]': finished 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T20:21:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/961955494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]: dispatch 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14a75dc5-6809-4c9f-9222-8e6a71c62a82"}]': finished 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T20:21:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:17.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:17 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/1627451832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:17.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:17 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/1627451832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:17.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:17 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/1627451832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:18.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:18 smithi136 ceph-mon[48073]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:18.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:18 smithi098 ceph-mon[43462]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:18 smithi169 ceph-mon[48529]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:20.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:20 smithi169 ceph-mon[48529]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:20 smithi098 ceph-mon[43462]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:20.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:20 smithi136 ceph-mon[48073]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:22 smithi169 ceph-mon[48529]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T20:21:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:22 smithi136 ceph-mon[48073]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T20:21:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:22.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:22 smithi098 ceph-mon[43462]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:22.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T20:21:22.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:23 smithi169 ceph-mon[48529]: Deploying daemon osd.2 on smithi169 2024-03-18T20:21:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:23 smithi136 ceph-mon[48073]: Deploying daemon osd.2 on smithi169 2024-03-18T20:21:23.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:23 smithi098 ceph-mon[43462]: Deploying daemon osd.2 on smithi169 2024-03-18T20:21:24.638 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:24 smithi169 ceph-mon[48529]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:24 smithi136 ceph-mon[48073]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:24.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:24 smithi098 ceph-mon[43462]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:26.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:26 smithi169 ceph-mon[48529]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:26.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:26.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:26.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:26 smithi136 ceph-mon[48073]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:27.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:26 smithi098 ceph-mon[43462]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:27.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:27.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:27.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:27.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.881+0000 7f70537fe700 1 -- 172.21.15.169:0/2745309747 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f705c0611f0 con 0x7f70480a1110 2024-03-18T20:21:27.887 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 2 on host 'smithi169' 2024-03-18T20:21:27.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 msgr2=0x7f70480a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 0x7f70480a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7054005fd0 tx=0x7f7054005dd0 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 msgr2=0x7f705c19e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f704c007a60 tx=0x7f704c004d30 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 shutdown_connections 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f705c101890 0x7f705c19e310 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70480a1110 0x7f70480a35d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c0ffcf0 0x7f705c196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 --2- 172.21.15.169:0/2745309747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c100ef0 0x7f705c1972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 >> 172.21.15.169:0/2745309747 conn(0x7f705c0fb2b0 msgr2=0x7f705c104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:27.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 shutdown_connections 2024-03-18T20:21:27.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:21:27.884+0000 7f7060ac1700 1 -- 172.21.15.169:0/2745309747 wait complete. 2024-03-18T20:21:28.267 DEBUG:teuthology.orchestra.run.smithi169:osd.2> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T20:21:28.275 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-18T20:21:28.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T20:21:28.307 INFO:journalctl@ceph.osd.2.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:21:28.511 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:28.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:28.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:28.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011157 s, 940 MB/s 2024-03-18T20:21:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-18T20:21:30.061 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:21:29.673+0000 7ff735144640 -1 osd.2 0 log_to_monitors true 2024-03-18T20:21:30.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-18T20:21:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:30.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:30.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:30.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:30.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 -- 172.21.15.98:0/4149544552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858101890 msgr2=0x7f6858108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 --2- 172.21.15.98:0/4149544552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858101890 0x7f6858108140 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f685000b0d0 tx=0x7f685000b3e0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 -- 172.21.15.98:0/4149544552 shutdown_connections 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 --2- 172.21.15.98:0/4149544552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858101890 0x7f6858108140 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 --2- 172.21.15.98:0/4149544552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68580ffcf0 0x7f6858100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 --2- 172.21.15.98:0/4149544552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6858100ef0 0x7f6858101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 -- 172.21.15.98:0/4149544552 >> 172.21.15.98:0/4149544552 conn(0x7f68580fb2b0 msgr2=0x7f68580fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 -- 172.21.15.98:0/4149544552 shutdown_connections 2024-03-18T20:21:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.313+0000 7f685c930700 1 -- 172.21.15.98:0/4149544552 wait complete. 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 Processor -- start 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 -- start start 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68580ffcf0 0x7f6858195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 0x7f685819d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685819d820 con 0x7f6858100ef0 2024-03-18T20:21:31.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685819d960 con 0x7f68580ffcf0 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f685c930700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685819db20 con 0x7f6858101890 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f6855d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.314+0000 7f6855d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34642/0 (socket says 172.21.15.98:34642) 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 -- 172.21.15.98:0/2430440892 learned_addr learned my addr 172.21.15.98:0/2430440892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f685659c700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68580ffcf0 0x7f6858195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6856d9d700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 0x7f685819d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 -- 172.21.15.98:0/2430440892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 msgr2=0x7f685819d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 0x7f685819d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 -- 172.21.15.98:0/2430440892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68580ffcf0 msgr2=0x7f6858195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68580ffcf0 0x7f6858195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 -- 172.21.15.98:0/2430440892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850009d00 con 0x7f6858100ef0 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6856d9d700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 0x7f685819d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:21:31.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f6855d9b700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f684800b530 tx=0x7f684800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848011830 con 0x7f6858100ef0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6848011e20 con 0x7f6858100ef0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848010dd0 con 0x7f6858100ef0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685819de00 con 0x7f6858100ef0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.315+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685819e2a0 con 0x7f6858100ef0 2024-03-18T20:21:31.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.316+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685804eaf0 con 0x7f6858100ef0 2024-03-18T20:21:31.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.317+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7f6848011990 con 0x7f6858100ef0 2024-03-18T20:21:31.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.318+0000 7f68477fe700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 0x7f68400a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:31.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.318+0000 7f685659c700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 0x7f68400a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:31.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.318+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2445+0+0 (secure 0 0 0) 0x7f68480cab90 con 0x7f6858100ef0 2024-03-18T20:21:31.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.318+0000 7f685659c700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 0x7f68400a35d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f684c006fd0 tx=0x7f684c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:31.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.322+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f685804eaf0 con 0x7f6858100ef0 2024-03-18T20:21:31.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:31.523+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f685804ea80 con 0x7f68400a1110 2024-03-18T20:21:31.884 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:21:31.601+0000 7ff72a7a8700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T20:21:31.884 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:21:31.606+0000 7ff72559b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:21:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi169 2024-03-18T20:21:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi169 to 17114M 2024-03-18T20:21:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T20:21:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:31.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi169 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi169 to 17114M 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:31.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:31.889 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi169 2024-03-18T20:21:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi169 to 17114M 2024-03-18T20:21:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T20:21:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:21:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T20:21:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:32.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:34.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:21:34.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:21:34.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] boot 2024-03-18T20:21:34.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T20:21:34.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:34.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]: dispatch 2024-03-18T20:21:34.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]': finished 2024-03-18T20:21:34.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: osdmap e19: 4 total, 3 up, 4 in 2024-03-18T20:21:34.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:33 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:34.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] boot 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]: dispatch 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]': finished 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: osdmap e19: 4 total, 3 up, 4 in 2024-03-18T20:21:34.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:33 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: osd.2 [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] boot 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]: dispatch 2024-03-18T20:21:34.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/363947474' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc4d0794-a030-42e7-8cdd-6e2c636e56ea"}]': finished 2024-03-18T20:21:34.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: osdmap e19: 4 total, 3 up, 4 in 2024-03-18T20:21:34.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:33 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1920262246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: osdmap e20: 4 total, 3 up, 4 in 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:35.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1920262246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: osdmap e20: 4 total, 3 up, 4 in 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-18T20:21:35.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1920262246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:35.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:35.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: osdmap e20: 4 total, 3 up, 4 in 2024-03-18T20:21:35.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:35.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T20:21:35.943 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:35.943 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:35.943 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:35.943 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69429]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:36.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69616]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:36.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69616]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:36.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69616]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:36.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:21:35 smithi098 sudo[69616]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:36.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:36.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:36 smithi136 ceph-mon[48073]: osdmap e21: 4 total, 3 up, 4 in 2024-03-18T20:21:36.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:36.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:36 smithi136 ceph-mon[48073]: pgmap v66: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:36.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58638]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58638]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58638]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58641]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58641]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:36.492 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:21:36 smithi136 sudo[58641]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:36.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:36.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:36 smithi169 ceph-mon[48529]: osdmap e21: 4 total, 3 up, 4 in 2024-03-18T20:21:36.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:36.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:36 smithi169 ceph-mon[48529]: pgmap v66: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:36.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T20:21:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:36 smithi098 ceph-mon[43462]: osdmap e21: 4 total, 3 up, 4 in 2024-03-18T20:21:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:36 smithi098 ceph-mon[43462]: pgmap v66: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:36.974+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6848095ec0 con 0x7f6858100ef0 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59014]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:21:36 smithi169 sudo[59017]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:37.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:37.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:37.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70156]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:37.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70156]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70200]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 sudo[70200]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 ceph-mon[43462]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T20:21:37.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:37.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:37.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 ceph-mon[43462]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-03-18T20:21:37.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 ceph-mon[48529]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 ceph-mon[48529]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:37.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59020]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:37 smithi136 ceph-mon[48073]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T20:21:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:37 smithi136 ceph-mon[48073]: mgrmap e16: x(active, since 111s), standbys: z, y 2024-03-18T20:21:37.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:38.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:38.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:38.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59023]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:38.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:37 smithi169 sudo[59023]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58644]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58644]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58647]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58647]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58647]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T20:21:38.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 sudo[58647]: pam_unix(sudo:session): session closed for user root 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: pgmap v68: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:38 smithi098 ceph-mon[43462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: pgmap v68: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:38 smithi169 ceph-mon[48529]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: pgmap v68: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T20:21:38.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:38 smithi136 ceph-mon[48073]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T20:21:39.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T20:21:39.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:39.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T20:21:39.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T20:21:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:40.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:40 smithi098 ceph-mon[43462]: Deploying daemon osd.3 on smithi098 2024-03-18T20:21:40.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:40 smithi098 ceph-mon[43462]: pgmap v69: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:40.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:40 smithi169 ceph-mon[48529]: Deploying daemon osd.3 on smithi098 2024-03-18T20:21:40.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:40 smithi169 ceph-mon[48529]: pgmap v69: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:40 smithi136 ceph-mon[48073]: Deploying daemon osd.3 on smithi098 2024-03-18T20:21:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:40 smithi136 ceph-mon[48073]: pgmap v69: 1 pgs: 1 active+clean; 449 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:42.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:42 smithi098 ceph-mon[43462]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 337 KiB/s wr, 24 op/s 2024-03-18T20:21:42.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:42 smithi169 ceph-mon[48529]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 337 KiB/s wr, 24 op/s 2024-03-18T20:21:42.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:42 smithi136 ceph-mon[48073]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 337 KiB/s wr, 24 op/s 2024-03-18T20:21:43.874 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:43.874 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:43.874 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:43.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:43.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:43.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.766+0000 7f68477fe700 1 -- 172.21.15.98:0/2430440892 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f685804ea80 con 0x7f68400a1110 2024-03-18T20:21:44.770 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 msgr2=0x7f68400a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 0x7f68400a35d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f684c006fd0 tx=0x7f684c008040 comp rx=0 tx=0).stop 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 msgr2=0x7f6858196230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f684800b530 tx=0x7f684800b8f0 comp rx=0 tx=0).stop 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 shutdown_connections 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6858101890 0x7f685819d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f68400a1110 0x7f68400a35d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68580ffcf0 0x7f6858195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 --2- 172.21.15.98:0/2430440892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6858100ef0 0x7f6858196230 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 >> 172.21.15.98:0/2430440892 conn(0x7f68580fb2b0 msgr2=0x7f6858104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 shutdown_connections 2024-03-18T20:21:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:21:44.768+0000 7f685c930700 1 -- 172.21.15.98:0/2430440892 wait complete. 2024-03-18T20:21:45.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 253 KiB/s wr, 18 op/s 2024-03-18T20:21:45.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 253 KiB/s wr, 18 op/s 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.149 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T20:21:45.151 INFO:tasks.cephadm:Deploying osd.4 on smithi136 with /dev/vg_nvme/lv_3... 2024-03-18T20:21:45.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T20:21:45.183 INFO:journalctl@ceph.osd.3.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 253 KiB/s wr, 18 op/s 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:45.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:21:45.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:45.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:46.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:46.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:46.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:46.205 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:46.205 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:46.205 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:21:46.745 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T20:21:46.745 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T20:21:46.746 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-18T20:21:46.746 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-18T20:21:46.746 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00856083 s, 1.2 GB/s 2024-03-18T20:21:46.746 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-18T20:21:47.082 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:46 smithi136 ceph-mon[48073]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.7 KiB/s rd, 202 KiB/s wr, 14 op/s 2024-03-18T20:21:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:46 smithi098 ceph-mon[43462]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.7 KiB/s rd, 202 KiB/s wr, 14 op/s 2024-03-18T20:21:47.093 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:21:46 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:21:46.763+0000 7fbcbfda9640 -1 osd.3 0 log_to_monitors true 2024-03-18T20:21:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:46 smithi169 ceph-mon[48529]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.7 KiB/s rd, 202 KiB/s wr, 14 op/s 2024-03-18T20:21:47.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-03-18T20:21:47.429 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:21:47.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:47.887 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:48.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:21:48.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 -- 172.21.15.136:0/2894607025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98101fe0 msgr2=0x7fee98102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2894607025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98101fe0 0x7fee98102440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee880099c0 tx=0x7fee88009cd0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 -- 172.21.15.136:0/2894607025 shutdown_connections 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2894607025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 0x7fee98109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2894607025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98100de0 0x7fee98101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2894607025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98101fe0 0x7fee98102440 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 -- 172.21.15.136:0/2894607025 >> 172.21.15.136:0/2894607025 conn(0x7fee980fc380 msgr2=0x7fee980fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:21:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.428+0000 7fee9ca37700 1 -- 172.21.15.136:0/2894607025 shutdown_connections 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 -- 172.21.15.136:0/2894607025 wait complete. 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 Processor -- start 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 -- start start 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98100de0 0x7fee98074ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 0x7fee980736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee98073dc0 con 0x7fee98100de0 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee98073f30 con 0x7fee98101fe0 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.429+0000 7fee9ca37700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee981a0ca0 con 0x7fee98102980 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36114/0 (socket says 172.21.15.136:36114) 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 -- 172.21.15.136:0/2570706250 learned_addr learned my addr 172.21.15.136:0/2570706250 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee9659c700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98100de0 0x7fee98074ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee96d9d700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 0x7fee980736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 -- 172.21.15.136:0/2570706250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 msgr2=0x7fee980736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 0x7fee980736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 -- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98100de0 msgr2=0x7fee98074ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98100de0 0x7fee98074ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 -- 172.21.15.136:0/2570706250 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c009520 con 0x7fee98101fe0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee95d9b700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fee88006010 tx=0x7fee8800f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8801d070 con 0x7fee98101fe0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee880095f0 con 0x7fee98101fe0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fee880054c0 con 0x7fee98101fe0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88004c00 con 0x7fee98101fe0 2024-03-18T20:21:48.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.430+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee981a1180 con 0x7fee98101fe0 2024-03-18T20:21:48.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.431+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee98108330 con 0x7fee98101fe0 2024-03-18T20:21:48.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.432+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fee8800fcc0 con 0x7fee98101fe0 2024-03-18T20:21:48.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.433+0000 7fee877fe700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 0x7fee800a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:21:48.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.433+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3373+0+0 (secure 0 0 0) 0x7fee880cc410 con 0x7fee98101fe0 2024-03-18T20:21:48.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.433+0000 7fee9659c700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 0x7fee800a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:21:48.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.433+0000 7fee9659c700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 0x7fee800a35e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fee98101e40 tx=0x7fee8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:21:48.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.437+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee88096ee0 con 0x7fee98101fe0 2024-03-18T20:21:48.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:21:48.637+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fee980611f0 con 0x7fee800a1120 2024-03-18T20:21:49.093 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:21:48.776+0000 7fbcb540d700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T20:21:49.093 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:21:48.782+0000 7fbcb1202700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:21:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi098 2024-03-18T20:21:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi098 to 8557M 2024-03-18T20:21:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 181 KiB/s wr, 13 op/s 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:49.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi098 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi098 to 8557M 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 181 KiB/s wr, 13 op/s 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:49.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi098 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi098 to 8557M 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 181 KiB/s wr, 13 op/s 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:21:49.184 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:21:49.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:21:49.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T20:21:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:50.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:50.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:21:50.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:21:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:21:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: from='osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:21:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T20:21:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:51.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:51.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] boot 2024-03-18T20:21:51.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T20:21:51.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:51.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/3830419888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]': finished 2024-03-18T20:21:51.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: osdmap e26: 5 total, 4 up, 5 in 2024-03-18T20:21:51.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:51.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:51.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] boot 2024-03-18T20:21:51.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/3830419888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]': finished 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: osdmap e26: 5 total, 4 up, 5 in 2024-03-18T20:21:51.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:51.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-18T20:21:51.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: osd.3 [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] boot 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/3830419888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]: dispatch 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fde25b-a451-4031-8795-9a05f0f2a29d"}]': finished 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: osdmap e26: 5 total, 4 up, 5 in 2024-03-18T20:21:51.244 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:52.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:51 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/4287934807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:52.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:51 smithi098 ceph-mon[43462]: osdmap e27: 5 total, 4 up, 5 in 2024-03-18T20:21:52.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:51 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:51 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/4287934807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:51 smithi169 ceph-mon[48529]: osdmap e27: 5 total, 4 up, 5 in 2024-03-18T20:21:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:51 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:51 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/4287934807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:21:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:51 smithi136 ceph-mon[48073]: osdmap e27: 5 total, 4 up, 5 in 2024-03-18T20:21:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:51 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:52 smithi098 ceph-mon[43462]: pgmap v80: 1 pgs: 1 remapped+peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:52 smithi098 ceph-mon[43462]: osdmap e28: 5 total, 4 up, 5 in 2024-03-18T20:21:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:52 smithi169 ceph-mon[48529]: pgmap v80: 1 pgs: 1 remapped+peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:52 smithi169 ceph-mon[48529]: osdmap e28: 5 total, 4 up, 5 in 2024-03-18T20:21:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:52 smithi136 ceph-mon[48073]: pgmap v80: 1 pgs: 1 remapped+peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:52 smithi136 ceph-mon[48073]: osdmap e28: 5 total, 4 up, 5 in 2024-03-18T20:21:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:54.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:54 smithi136 ceph-mon[48073]: osdmap e29: 5 total, 4 up, 5 in 2024-03-18T20:21:54.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:54.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:54 smithi136 ceph-mon[48073]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:54.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:54 smithi098 ceph-mon[43462]: osdmap e29: 5 total, 4 up, 5 in 2024-03-18T20:21:54.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:54.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:54 smithi098 ceph-mon[43462]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:54.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:54 smithi169 ceph-mon[48529]: osdmap e29: 5 total, 4 up, 5 in 2024-03-18T20:21:54.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:21:54.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:54 smithi169 ceph-mon[48529]: pgmap v83: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:21:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:56 smithi169 ceph-mon[48529]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T20:21:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T20:21:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:56.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:56 smithi136 ceph-mon[48073]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T20:21:56.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T20:21:56.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:56 smithi098 ceph-mon[43462]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T20:21:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T20:21:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:21:57.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:57 smithi136 ceph-mon[48073]: Deploying daemon osd.4 on smithi136 2024-03-18T20:21:57.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:57 smithi169 ceph-mon[48529]: Deploying daemon osd.4 on smithi136 2024-03-18T20:21:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:57 smithi098 ceph-mon[43462]: Deploying daemon osd.4 on smithi136 2024-03-18T20:21:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:21:58 smithi169 ceph-mon[48529]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-18T20:21:58.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:21:58 smithi136 ceph-mon[48073]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-18T20:21:59.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:21:58 smithi098 ceph-mon[43462]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-18T20:22:00.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:00 smithi136 ceph-mon[48073]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:00.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:00 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:00.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:00 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:00.726 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:00 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:00 smithi169 ceph-mon[48529]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:00 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:00 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:00 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:01.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:00 smithi098 ceph-mon[43462]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:01.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:00 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:01.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:00 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:01.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:00 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:02.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.096+0000 7fee877fe700 1 -- 172.21.15.136:0/2570706250 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fee980611f0 con 0x7fee800a1120 2024-03-18T20:22:02.103 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 4 on host 'smithi136' 2024-03-18T20:22:02.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 msgr2=0x7fee800a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 0x7fee800a35e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fee98101e40 tx=0x7fee8c009260 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 msgr2=0x7fee980731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fee88006010 tx=0x7fee8800f740 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 shutdown_connections 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee98102980 0x7fee980736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee98100de0 0x7fee98074ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fee800a1120 0x7fee800a35e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 --2- 172.21.15.136:0/2570706250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98101fe0 0x7fee980731a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 >> 172.21.15.136:0/2570706250 conn(0x7fee980fc380 msgr2=0x7fee98105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 shutdown_connections 2024-03-18T20:22:02.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:02.099+0000 7fee9ca37700 1 -- 172.21.15.136:0/2570706250 wait complete. 2024-03-18T20:22:02.478 DEBUG:teuthology.orchestra.run.smithi136:osd.4> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T20:22:02.480 INFO:tasks.cephadm:Deploying osd.5 on smithi169 with /dev/vg_nvme/lv_3... 2024-03-18T20:22:02.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T20:22:02.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:02.508 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:02.509 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.509 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T20:22:02.509 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.509 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.509 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:02.513 INFO:journalctl@ceph.osd.4.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T20:22:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:02.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114258 s, 918 MB/s 2024-03-18T20:22:04.072 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-18T20:22:04.241 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:22:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:22:03.892+0000 7f65d37ae640 -1 osd.4 0 log_to_monitors true 2024-03-18T20:22:04.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-03-18T20:22:04.683 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:22:04.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:04 smithi169 ceph-mon[48529]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:04.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:04 smithi169 ceph-mon[48529]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:04.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:04 smithi169 ceph-mon[48529]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:04 smithi136 ceph-mon[48073]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:04 smithi136 ceph-mon[48073]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:04 smithi136 ceph-mon[48073]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:05.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:04 smithi098 ceph-mon[43462]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:05.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:04 smithi098 ceph-mon[43462]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:05.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:04 smithi098 ceph-mon[43462]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 -- 172.21.15.169:0/3370093531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 msgr2=0x7f8734103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 --2- 172.21.15.169:0/3370093531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734103c70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8728009990 tx=0x7f8728009ca0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 -- 172.21.15.169:0/3370093531 shutdown_connections 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 --2- 172.21.15.169:0/3370093531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f8734108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 --2- 172.21.15.169:0/3370093531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734103c70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 --2- 172.21.15.169:0/3370093531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 0x7f87341065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 -- 172.21.15.169:0/3370093531 >> 172.21.15.169:0/3370093531 conn(0x7f87340fb2b0 msgr2=0x7f87340fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 -- 172.21.15.169:0/3370093531 shutdown_connections 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.678+0000 7f873aae3700 1 -- 172.21.15.169:0/3370093531 wait complete. 2024-03-18T20:22:05.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 Processor -- start 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 -- start start 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 0x7f87341971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f873419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873419e7e0 con 0x7f8734106ae0 2024-03-18T20:22:05.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873419e920 con 0x7f87341041b0 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873aae3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873419eae0 con 0x7f8734101880 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873887f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873887f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41506/0 (socket says 172.21.15.169:41506) 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.679+0000 7f873887f700 1 -- 172.21.15.169:0/1685818191 learned_addr learned my addr 172.21.15.169:0/1685818191 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8733fff700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 0x7f87341971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8739080700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f873419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 -- 172.21.15.169:0/1685818191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 msgr2=0x7f87341971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 0x7f87341971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 -- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 msgr2=0x7f873419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f873419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 -- 172.21.15.169:0/1685818191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280095f0 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8739080700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f873419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873887f700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f87280098e0 tx=0x7f8728003a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872801d070 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f872800bd90 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873419ed60 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800f460 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.680+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873419f180 con 0x7f8734101880 2024-03-18T20:22:05.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.681+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87340fce30 con 0x7f8734101880 2024-03-18T20:22:05.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.682+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8728022470 con 0x7f8734101880 2024-03-18T20:22:05.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.683+0000 7f8731ffb700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 0x7f87240a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:05.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.683+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3821+0+0 (secure 0 0 0) 0x7f87280cccf0 con 0x7f8734101880 2024-03-18T20:22:05.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.683+0000 7f8733fff700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 0x7f87240a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:05.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.684+0000 7f8733fff700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 0x7f87240a36e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8734198170 tx=0x7f8720009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:05.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.688+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8728097600 con 0x7f8734101880 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi136 2024-03-18T20:22:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi136 to 8556M 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:05.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:05 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:05.887+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f87340611f0 con 0x7f87240a1220 2024-03-18T20:22:05.991 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:22:05.612+0000 7f65c8e12700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T20:22:05.991 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:22:05.619+0000 7f65c2c03700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi136 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi136 to 8556M 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:05.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:05 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi136 2024-03-18T20:22:06.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi136 to 8556M 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:06.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:05 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T20:22:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:06.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:06.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:06.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:06.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:06.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:07.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] boot 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: osdmap e32: 5 total, 5 up, 5 in 2024-03-18T20:22:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:22:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:22:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] boot 2024-03-18T20:22:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: osdmap e32: 5 total, 5 up, 5 in 2024-03-18T20:22:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: osd.4 [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] boot 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: osdmap e32: 5 total, 5 up, 5 in 2024-03-18T20:22:08.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T20:22:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T20:22:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/1899461138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:08.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]': finished 2024-03-18T20:22:08.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: osdmap e34: 6 total, 5 up, 6 in 2024-03-18T20:22:08.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:08.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:08 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/13730985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/1899461138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]': finished 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: osdmap e34: 6 total, 5 up, 6 in 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:08 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/13730985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-18T20:22:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T20:22:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/1899461138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:09.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]: dispatch 2024-03-18T20:22:09.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62b9ea73-b07c-4aa1-8d5d-57fdf0204185"}]': finished 2024-03-18T20:22:09.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: osdmap e34: 6 total, 5 up, 6 in 2024-03-18T20:22:09.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:09.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:08 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/13730985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:10 smithi136 ceph-mon[48073]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:11.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:10 smithi098 ceph-mon[43462]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:11.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:10 smithi169 ceph-mon[48529]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:12.551 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:12 smithi169 ceph-mon[48529]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:12.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:12 smithi136 ceph-mon[48073]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:12 smithi098 ceph-mon[43462]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:14.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:14 smithi169 ceph-mon[48529]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:14.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T20:22:14.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:14.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:14 smithi169 ceph-mon[48529]: Deploying daemon osd.5 on smithi169 2024-03-18T20:22:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:14 smithi136 ceph-mon[48073]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T20:22:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:14 smithi136 ceph-mon[48073]: Deploying daemon osd.5 on smithi169 2024-03-18T20:22:15.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:14 smithi098 ceph-mon[43462]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:15.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T20:22:15.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:15.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:14 smithi098 ceph-mon[43462]: Deploying daemon osd.5 on smithi169 2024-03-18T20:22:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:16 smithi169 ceph-mon[48529]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:16 smithi136 ceph-mon[48073]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:16 smithi098 ceph-mon[43462]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:18.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:18 smithi169 ceph-mon[48529]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:18.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:18 smithi098 ceph-mon[43462]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:18 smithi136 ceph-mon[48073]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:18.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:19.898 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 5 on host 'smithi169' 2024-03-18T20:22:19.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.893+0000 7f8731ffb700 1 -- 172.21.15.169:0/1685818191 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f87340611f0 con 0x7f87240a1220 2024-03-18T20:22:19.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 msgr2=0x7f87240a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:19.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 0x7f87240a36e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8734198170 tx=0x7f8720009190 comp rx=0 tx=0).stop 2024-03-18T20:22:19.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 msgr2=0x7f8734196cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:19.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f87280098e0 tx=0x7f8728003a50 comp rx=0 tx=0).stop 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 shutdown_connections 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8734106ae0 0x7f873419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8734101880 0x7f8734196cb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87240a1220 0x7f87240a36e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 --2- 172.21.15.169:0/1685818191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87341041b0 0x7f87341971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 >> 172.21.15.169:0/1685818191 conn(0x7f87340fb2b0 msgr2=0x7f8734102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.895+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 shutdown_connections 2024-03-18T20:22:19.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:19.896+0000 7f873aae3700 1 -- 172.21.15.169:0/1685818191 wait complete. 2024-03-18T20:22:20.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:20.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:20.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:20.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:20.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.285 DEBUG:teuthology.orchestra.run.smithi169:osd.5> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T20:22:20.287 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-18T20:22:20.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T20:22:20.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:20.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:20.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.322 INFO:journalctl@ceph.osd.5.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:22:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:22:20.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:20 smithi098 ceph-mon[43462]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:20.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:20.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:20 smithi169 ceph-mon[48529]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:20 smithi136 ceph-mon[48073]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112371 s, 933 MB/s 2024-03-18T20:22:21.832 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-18T20:22:22.056 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:22:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:22:21.634+0000 7f23d4d3c640 -1 osd.5 0 log_to_monitors true 2024-03-18T20:22:22.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-18T20:22:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:22.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T20:22:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 -- 172.21.15.98:0/3000893639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 msgr2=0x7faff0109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 --2- 172.21.15.98:0/3000893639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff0109230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fafe800b0d0 tx=0x7fafe800b3e0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 -- 172.21.15.98:0/3000893639 shutdown_connections 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 --2- 172.21.15.98:0/3000893639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff0109230 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 --2- 172.21.15.98:0/3000893639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0100de0 0x7faff0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.489+0000 7faff63b7700 1 --2- 172.21.15.98:0/3000893639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0101fe0 0x7faff0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- 172.21.15.98:0/3000893639 >> 172.21.15.98:0/3000893639 conn(0x7faff00fc380 msgr2=0x7faff00fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- 172.21.15.98:0/3000893639 shutdown_connections 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- 172.21.15.98:0/3000893639 wait complete. 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 Processor -- start 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- start start 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0100de0 0x7faff0196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0101fe0 0x7faff0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff019e830 con 0x7faff0102980 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff019e970 con 0x7faff0101fe0 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.490+0000 7faff63b7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff019eb30 con 0x7faff0100de0 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57256/0 (socket says 172.21.15.98:57256) 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 -- 172.21.15.98:0/1412963034 learned_addr learned my addr 172.21.15.98:0/1412963034 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7fafeffff700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0100de0 0x7faff0196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7fafef7fe700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0101fe0 0x7faff0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 -- 172.21.15.98:0/1412963034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0100de0 msgr2=0x7faff0196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0100de0 0x7faff0196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 -- 172.21.15.98:0/1412963034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0101fe0 msgr2=0x7faff0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0101fe0 0x7faff0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 -- 172.21.15.98:0/1412963034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8009d00 con 0x7faff0102980 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff4954700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fafe8007b60 tx=0x7fafe8009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe800e040 con 0x7faff0102980 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff019edb0 con 0x7faff0102980 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fafe801d480 con 0x7faff0102980 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe801b6b0 con 0x7faff0102980 2024-03-18T20:22:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.491+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff019f210 con 0x7faff0102980 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.493+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff01082c0 con 0x7faff0102980 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.493+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fafe801d5f0 con 0x7faff0102980 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.494+0000 7fafed7fa700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 0x7fafd80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.494+0000 7fafeffff700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 0x7fafd80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.494+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4253+0+0 (secure 0 0 0) 0x7fafe80cd7b0 con 0x7faff0102980 2024-03-18T20:22:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.494+0000 7fafeffff700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 0x7fafd80a3660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faff00fe840 tx=0x7fafe000b5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.498+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fafe8097f10 con 0x7faff0102980 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi169 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi169 to 8557M 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:23.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:23 smithi098 ceph-mon[43462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:23.634 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:22:23.387+0000 7f23ca3a0700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T20:22:23.634 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:22:23.395+0000 7f23c4992700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi169 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi169 to 8557M 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T20:22:23.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:23.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:23.635 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:23 smithi169 ceph-mon[48529]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:23.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:23.702+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faff002d130 con 0x7fafd80a11a0 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi169 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi169 to 8557M 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: from='osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:23.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:23 smithi136 ceph-mon[48073]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:22:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:22:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T20:22:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:24.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:22:24.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T20:22:24.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:24.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:24.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:24.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:24.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:22:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T20:22:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-18T20:22:24.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:24.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:24.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:24.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:25.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:22:25.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:22:25.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:25.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] boot 2024-03-18T20:22:25.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: osdmap e37: 6 total, 6 up, 6 in 2024-03-18T20:22:25.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:25 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] boot 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: osdmap e37: 6 total, 6 up, 6 in 2024-03-18T20:22:25.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:25 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: osd.5 [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] boot 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: osdmap e37: 6 total, 6 up, 6 in 2024-03-18T20:22:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:25 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T20:22:26.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T20:22:26.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:26.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]: dispatch 2024-03-18T20:22:26.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]': finished 2024-03-18T20:22:26.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: osdmap e39: 7 total, 6 up, 7 in 2024-03-18T20:22:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1785293435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: osdmap e40: 7 total, 6 up, 7 in 2024-03-18T20:22:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]: dispatch 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]': finished 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: osdmap e39: 7 total, 6 up, 7 in 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1785293435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: osdmap e40: 7 total, 6 up, 7 in 2024-03-18T20:22:26.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]: dispatch 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/243760670' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1"}]': finished 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: osdmap e39: 7 total, 6 up, 7 in 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1785293435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: osdmap e40: 7 total, 6 up, 7 in 2024-03-18T20:22:26.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:28.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:28 smithi136 ceph-mon[48073]: osdmap e41: 7 total, 6 up, 7 in 2024-03-18T20:22:28.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:28.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:28 smithi136 ceph-mon[48073]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:28.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:28 smithi098 ceph-mon[43462]: osdmap e41: 7 total, 6 up, 7 in 2024-03-18T20:22:28.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:28.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:28 smithi098 ceph-mon[43462]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:28.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:28 smithi169 ceph-mon[48529]: osdmap e41: 7 total, 6 up, 7 in 2024-03-18T20:22:28.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:28.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:28 smithi169 ceph-mon[48529]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:30.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:30 smithi169 ceph-mon[48529]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:30 smithi136 ceph-mon[48073]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:31.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:30 smithi098 ceph-mon[43462]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T20:22:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:32 smithi136 ceph-mon[48073]: Deploying daemon osd.6 on smithi098 2024-03-18T20:22:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:32 smithi136 ceph-mon[48073]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:22:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T20:22:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:32 smithi098 ceph-mon[43462]: Deploying daemon osd.6 on smithi098 2024-03-18T20:22:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:32 smithi098 ceph-mon[43462]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:22:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T20:22:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:32 smithi169 ceph-mon[48529]: Deploying daemon osd.6 on smithi098 2024-03-18T20:22:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:32 smithi169 ceph-mon[48529]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:22:34.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:34 smithi098 ceph-mon[43462]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:34 smithi169 ceph-mon[48529]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:34.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:34 smithi136 ceph-mon[48073]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:22:36.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:36.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:35 smithi098 ceph-mon[43462]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T20:22:36.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:36.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:35 smithi136 ceph-mon[48073]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T20:22:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:36.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:35 smithi169 ceph-mon[48529]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T20:22:37.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.045+0000 7fafed7fa700 1 -- 172.21.15.98:0/1412963034 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faff002d130 con 0x7fafd80a11a0 2024-03-18T20:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.047+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 msgr2=0x7fafd80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.047+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 0x7fafd80a3660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faff00fe840 tx=0x7fafe000b5d0 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.047+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 msgr2=0x7faff019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.047+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fafe8007b60 tx=0x7fafe8009570 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 shutdown_connections 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faff0102980 0x7faff019e260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faff0100de0 0x7faff0196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafd80a11a0 0x7fafd80a3660 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 --2- 172.21.15.98:0/1412963034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff0101fe0 0x7faff0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 >> 172.21.15.98:0/1412963034 conn(0x7faff00fc380 msgr2=0x7faff0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 shutdown_connections 2024-03-18T20:22:37.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:22:37.048+0000 7faff63b7700 1 -- 172.21.15.98:0/1412963034 wait complete. 2024-03-18T20:22:37.464 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T20:22:37.466 INFO:tasks.cephadm:Deploying osd.7 on smithi136 with /dev/vg_nvme/lv_2... 2024-03-18T20:22:37.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T20:22:37.496 INFO:journalctl@ceph.osd.6.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:22:37.697 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:22:37.835 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.835 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.835 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:37.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:37.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:37.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:37.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:37.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:37.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:38.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:38 smithi136 ceph-mon[48073]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:39.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:38 smithi098 ceph-mon[43462]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:39.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:22:38 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:22:38.899+0000 7f3927ad6640 -1 osd.6 0 log_to_monitors true 2024-03-18T20:22:39.108 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T20:22:39.109 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T20:22:39.109 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-18T20:22:39.109 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-18T20:22:39.109 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0080894 s, 1.3 GB/s 2024-03-18T20:22:39.109 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-18T20:22:39.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:38 smithi169 ceph-mon[48529]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-18T20:22:39.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-03-18T20:22:39.764 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:39.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:39.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:39.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:40.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:40.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:40.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:40.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:40.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 -- 172.21.15.136:0/2866570295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310101ff0 msgr2=0x7fb310102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 --2- 172.21.15.136:0/2866570295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310101ff0 0x7fb310102450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb3040099c0 tx=0x7fb304009cd0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 -- 172.21.15.136:0/2866570295 shutdown_connections 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 --2- 172.21.15.136:0/2866570295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310102990 0x7fb310109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 --2- 172.21.15.136:0/2866570295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310100df0 0x7fb310101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 --2- 172.21.15.136:0/2866570295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310101ff0 0x7fb310102450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 -- 172.21.15.136:0/2866570295 >> 172.21.15.136:0/2866570295 conn(0x7fb3100fc390 msgr2=0x7fb3100fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 -- 172.21.15.136:0/2866570295 shutdown_connections 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.793+0000 7fb314f51700 1 -- 172.21.15.136:0/2866570295 wait complete. 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 Processor -- start 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 -- start start 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310100df0 0x7fb310074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310101ff0 0x7fb3100731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb310073e80 con 0x7fb310100df0 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb310073ff0 con 0x7fb310102990 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb314f51700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3101a0c50 con 0x7fb310101ff0 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:40.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45536/0 (socket says 172.21.15.136:45536) 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 -- 172.21.15.136:0/78443073 learned_addr learned my addr 172.21.15.136:0/78443073 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30e59c700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310100df0 0x7fb310074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 -- 172.21.15.136:0/78443073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310101ff0 msgr2=0x7fb3100731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30dd9b700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310101ff0 0x7fb3100731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310101ff0 0x7fb3100731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 -- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310100df0 msgr2=0x7fb310074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310100df0 0x7fb310074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 -- 172.21.15.136:0/78443073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3040095f0 con 0x7fb310102990 2024-03-18T20:22:40.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb30ed9d700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb308007ee0 tx=0x7fb30800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.794+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308017070 con 0x7fb310102990 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.795+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb30800f040 con 0x7fb310102990 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.795+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3101a0ee0 con 0x7fb310102990 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.795+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308005710 con 0x7fb310102990 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.795+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3101a1350 con 0x7fb310102990 2024-03-18T20:22:40.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.796+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3101082c0 con 0x7fb310102990 2024-03-18T20:22:40.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.797+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb308005870 con 0x7fb310102990 2024-03-18T20:22:40.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.798+0000 7fb2ff7fe700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 0x7fb2f80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:40.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.798+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4685+0+0 (secure 0 0 0) 0x7fb3080cb7d0 con 0x7fb310102990 2024-03-18T20:22:40.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.798+0000 7fb30e59c700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 0x7fb2f80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:40.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.799+0000 7fb30e59c700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 0x7fb2f80a3660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb310101e50 tx=0x7fb300007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:40.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:40.802+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb308095d90 con 0x7fb310102990 2024-03-18T20:22:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:41.001+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3100611f0 con 0x7fb2f80a11a0 2024-03-18T20:22:41.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:22:40.746+0000 7f391d13a700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T20:22:41.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:22:40.757+0000 7f391772c700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:22:41.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi098 2024-03-18T20:22:41.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi098 to 5704M 2024-03-18T20:22:41.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T20:22:41.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T20:22:41.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: osdmap e42: 7 total, 6 up, 7 in 2024-03-18T20:22:41.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:41.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:40 smithi098 ceph-mon[43462]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi098 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi098 to 5704M 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: osdmap e42: 7 total, 6 up, 7 in 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:40 smithi169 ceph-mon[48529]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi098 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi098 to 5704M 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: osdmap e42: 7 total, 6 up, 7 in 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:40 smithi136 ceph-mon[48073]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:42.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] boot 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: osdmap e44: 7 total, 7 up, 7 in 2024-03-18T20:22:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:22:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:22:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] boot 2024-03-18T20:22:43.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: osdmap e44: 7 total, 7 up, 7 in 2024-03-18T20:22:43.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: osd.6 [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] boot 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: osdmap e44: 7 total, 7 up, 7 in 2024-03-18T20:22:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: osdmap e45: 7 total, 7 up, 7 in 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/4255718892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]': finished 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: osdmap e46: 8 total, 7 up, 8 in 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:44.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:43 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/3859657511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: osdmap e45: 7 total, 7 up, 7 in 2024-03-18T20:22:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/4255718892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]': finished 2024-03-18T20:22:44.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: osdmap e46: 8 total, 7 up, 8 in 2024-03-18T20:22:44.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:44.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:43 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/3859657511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:44.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: osdmap e45: 7 total, 7 up, 7 in 2024-03-18T20:22:44.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/4255718892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]: dispatch 2024-03-18T20:22:44.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df65208c-eee4-455f-b0e0-9a42f261531e"}]': finished 2024-03-18T20:22:44.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: osdmap e46: 8 total, 7 up, 8 in 2024-03-18T20:22:44.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:44.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:43 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/3859657511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:22:45.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:44 smithi098 ceph-mon[43462]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:45.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:44 smithi169 ceph-mon[48529]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:45.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:44 smithi136 ceph-mon[48073]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:46.928 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:46 smithi136 ceph-mon[48073]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:46 smithi098 ceph-mon[43462]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:46 smithi169 ceph-mon[48529]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:48 smithi098 ceph-mon[43462]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:48 smithi169 ceph-mon[48529]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:49.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:48 smithi136 ceph-mon[48073]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:49.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T20:22:49.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:49.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:49 smithi136 ceph-mon[48073]: Deploying daemon osd.7 on smithi136 2024-03-18T20:22:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T20:22:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:50.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:49 smithi098 ceph-mon[43462]: Deploying daemon osd.7 on smithi136 2024-03-18T20:22:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T20:22:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:49 smithi169 ceph-mon[48529]: Deploying daemon osd.7 on smithi136 2024-03-18T20:22:50.936 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:50 smithi136 ceph-mon[48073]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:51.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:50 smithi098 ceph-mon[43462]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:51.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:50 smithi169 ceph-mon[48529]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:52.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:52 smithi136 ceph-mon[48073]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:52.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:52 smithi098 ceph-mon[43462]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:52.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:52 smithi169 ceph-mon[48529]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:53.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:53 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:53.788 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:53 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:53.788 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:53 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:54.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:53 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:53 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:53 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:53 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:53 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:53 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:54.959 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:54.959 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.959 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:54.959 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:54.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:54.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.053+0000 7fb2ff7fe700 1 -- 172.21.15.136:0/78443073 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb3100611f0 con 0x7fb2f80a11a0 2024-03-18T20:22:55.058 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 7 on host 'smithi136' 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 msgr2=0x7fb2f80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 0x7fb2f80a3660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb310101e50 tx=0x7fb300007400 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 msgr2=0x7fb3100736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb308007ee0 tx=0x7fb30800d3b0 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 shutdown_connections 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb2f80a11a0 0x7fb2f80a3660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb310102990 0x7fb3100736e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb310100df0 0x7fb310074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 --2- 172.21.15.136:0/78443073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb310101ff0 0x7fb3100731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 >> 172.21.15.136:0/78443073 conn(0x7fb3100fc390 msgr2=0x7fb310105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:55.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 shutdown_connections 2024-03-18T20:22:55.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:22:55.055+0000 7fb314f51700 1 -- 172.21.15.136:0/78443073 wait complete. 2024-03-18T20:22:55.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:55.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:55.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:55.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:55.411 DEBUG:teuthology.orchestra.run.smithi136:osd.7> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T20:22:55.413 INFO:tasks.cephadm:Deploying osd.8 on smithi169 with /dev/vg_nvme/lv_2... 2024-03-18T20:22:55.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T20:22:55.446 INFO:journalctl@ceph.osd.7.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:22:55.648 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:22:56.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:56 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:56.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:56 smithi136 ceph-mon[48073]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:56.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:56 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:56.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:56 smithi098 ceph-mon[43462]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:56.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:56.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:56 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:22:56.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:56 smithi169 ceph-mon[48529]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T20:22:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T20:22:56.955 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-18T20:22:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-18T20:22:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113479 s, 924 MB/s 2024-03-18T20:22:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-18T20:22:56.991 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:22:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:22:56.724+0000 7efd37d97640 -1 osd.7 0 log_to_monitors true 2024-03-18T20:22:57.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-03-18T20:22:57.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:57 smithi136 ceph-mon[48073]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:57.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:57 smithi136 ceph-mon[48073]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:57.539 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:22:57.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:57 smithi169 ceph-mon[48529]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:57.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:57 smithi169 ceph-mon[48529]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:57.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:57 smithi098 ceph-mon[43462]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:57.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:57 smithi098 ceph-mon[43462]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T20:22:58.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 -- 172.21.15.169:0/230752156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900101890 msgr2=0x7f3900108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:58.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 --2- 172.21.15.169:0/230752156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900101890 0x7f3900108140 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f38f800b0a0 tx=0x7f38f800b3b0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 -- 172.21.15.169:0/230752156 shutdown_connections 2024-03-18T20:22:58.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 --2- 172.21.15.169:0/230752156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900101890 0x7f3900108140 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 --2- 172.21.15.169:0/230752156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39000ffcf0 0x7f3900100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 --2- 172.21.15.169:0/230752156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3900100ef0 0x7f3900101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 -- 172.21.15.169:0/230752156 >> 172.21.15.169:0/230752156 conn(0x7f39000fb2b0 msgr2=0x7f39000fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 -- 172.21.15.169:0/230752156 shutdown_connections 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.559+0000 7f3905eef700 1 -- 172.21.15.169:0/230752156 wait complete. 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 Processor -- start 2024-03-18T20:22:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 -- start start 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39000ffcf0 0x7f3900198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3900101890 0x7f39001a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001a0ac0 con 0x7f39000ffcf0 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001a0c00 con 0x7f3900101890 2024-03-18T20:22:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f3905eef700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001a0d90 con 0x7f3900100ef0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f38feffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f38feffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53046/0 (socket says 172.21.15.169:53046) 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f38feffd700 1 -- 172.21.15.169:0/818471894 learned_addr learned my addr 172.21.15.169:0/818471894 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f38fffff700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3900101890 0x7f39001a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.560+0000 7f38ff7fe700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39000ffcf0 0x7f3900198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 -- 172.21.15.169:0/818471894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3900101890 msgr2=0x7f39001a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3900101890 0x7f39001a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 -- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39000ffcf0 msgr2=0x7f3900198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39000ffcf0 0x7f3900198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 -- 172.21.15.169:0/818471894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8009d00 con 0x7f3900100ef0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38feffd700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b560 tx=0x7f38f000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0011830 con 0x7f3900100ef0 2024-03-18T20:22:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f38f0011e20 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0010dd0 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39001a1070 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.561+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39001a1570 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.562+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390004ea80 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.563+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f38f0011990 con 0x7f3900100ef0 2024-03-18T20:22:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.563+0000 7f38fcff9700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 0x7f38e80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:22:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.564+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5117+0+0 (secure 0 0 0) 0x7f38f00cb720 con 0x7f3900100ef0 2024-03-18T20:22:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.564+0000 7f38ff7fe700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 0x7f38e80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:22:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.564+0000 7f38ff7fe700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 0x7f38e80a3660 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38f4005950 tx=0x7f38f40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:22:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.568+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f38f0095a90 con 0x7f3900100ef0 2024-03-18T20:22:58.741 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:22:58.399+0000 7efd2dbfc700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T20:22:58.742 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:22:58.413+0000 7efd251e8700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:22:58.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T20:22:58.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: osdmap e47: 8 total, 7 up, 8 in 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi136 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.743 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi136 to 5704M 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.744 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:58 smithi136 ceph-mon[48073]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:58.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:22:58.766+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f39001a17f0 con 0x7f38e80a11a0 2024-03-18T20:22:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: osdmap e47: 8 total, 7 up, 8 in 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi136 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi136 to 5704M 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:58.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:58.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:58 smithi098 ceph-mon[43462]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: osdmap e47: 8 total, 7 up, 8 in 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi136 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi136 to 5704M 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:22:58.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:58 smithi169 ceph-mon[48529]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:22:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:22:59 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:59.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:22:59 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:22:59.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:22:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:22:59 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:23:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:23:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] boot 2024-03-18T20:23:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: osdmap e49: 8 total, 8 up, 8 in 2024-03-18T20:23:00.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:23:00.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:00 smithi136 ceph-mon[48073]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] boot 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: osdmap e49: 8 total, 8 up, 8 in 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:23:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:00 smithi098 ceph-mon[43462]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:23:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:23:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:23:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: osd.7 [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] boot 2024-03-18T20:23:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: osdmap e49: 8 total, 8 up, 8 in 2024-03-18T20:23:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T20:23:00.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:00 smithi169 ceph-mon[48529]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-18T20:23:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: osdmap e50: 8 total, 8 up, 8 in 2024-03-18T20:23:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/3913406636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]': finished 2024-03-18T20:23:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: osdmap e51: 9 total, 8 up, 9 in 2024-03-18T20:23:01.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:01.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/3652280410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:01.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: osdmap e52: 9 total, 8 up, 9 in 2024-03-18T20:23:01.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:01 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: osdmap e50: 8 total, 8 up, 8 in 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/3913406636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]': finished 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: osdmap e51: 9 total, 8 up, 9 in 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/3652280410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: osdmap e52: 9 total, 8 up, 9 in 2024-03-18T20:23:01.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:01 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: osdmap e50: 8 total, 8 up, 8 in 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/3913406636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]: dispatch 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ff34b6f-b1af-4751-afda-9fa4bc03e881"}]': finished 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: osdmap e51: 9 total, 8 up, 9 in 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/3652280410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: osdmap e52: 9 total, 8 up, 9 in 2024-03-18T20:23:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:01 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:02 smithi136 ceph-mon[48073]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:02 smithi136 ceph-mon[48073]: osdmap e53: 9 total, 8 up, 9 in 2024-03-18T20:23:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:02.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:02 smithi098 ceph-mon[43462]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:02.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:02 smithi098 ceph-mon[43462]: osdmap e53: 9 total, 8 up, 9 in 2024-03-18T20:23:02.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:02.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:02 smithi169 ceph-mon[48529]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:02.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:02 smithi169 ceph-mon[48529]: osdmap e53: 9 total, 8 up, 9 in 2024-03-18T20:23:02.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:04.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:04 smithi169 ceph-mon[48529]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:04 smithi136 ceph-mon[48073]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:05.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:04 smithi098 ceph-mon[43462]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:06 smithi169 ceph-mon[48529]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T20:23:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:06 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:06 smithi136 ceph-mon[48073]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T20:23:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:06 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:06 smithi098 ceph-mon[43462]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T20:23:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:06 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:07.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:07 smithi169 ceph-mon[48529]: Deploying daemon osd.8 on smithi169 2024-03-18T20:23:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:07 smithi136 ceph-mon[48073]: Deploying daemon osd.8 on smithi169 2024-03-18T20:23:08.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:07 smithi098 ceph-mon[43462]: Deploying daemon osd.8 on smithi169 2024-03-18T20:23:08.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:08 smithi169 ceph-mon[48529]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:08 smithi136 ceph-mon[48073]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:08 smithi098 ceph-mon[43462]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:10.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:10 smithi169 ceph-mon[48529]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:10.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:10.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:10.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:10 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:10 smithi136 ceph-mon[48073]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:10 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:11.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:10 smithi098 ceph-mon[43462]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:11.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:11.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:11.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:10 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:12.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.568+0000 7f38fcff9700 1 -- 172.21.15.169:0/818471894 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f39001a17f0 con 0x7f38e80a11a0 2024-03-18T20:23:12.573 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 8 on host 'smithi169' 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 msgr2=0x7f38e80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 0x7f38e80a3660 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38f4005950 tx=0x7f38f40058e0 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 msgr2=0x7f39001994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b560 tx=0x7f38f000b870 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 shutdown_connections 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38e80a11a0 0x7f38e80a3660 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3900101890 0x7f39001a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39000ffcf0 0x7f3900198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 --2- 172.21.15.169:0/818471894 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3900100ef0 0x7f39001994c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 >> 172.21.15.169:0/818471894 conn(0x7f39000fb2b0 msgr2=0x7f39000fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 shutdown_connections 2024-03-18T20:23:12.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:12.570+0000 7f3905eef700 1 -- 172.21.15.169:0/818471894 wait complete. 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:12 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:12.749 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:12 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:12 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:12.966 DEBUG:teuthology.orchestra.run.smithi169:osd.8> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T20:23:12.969 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-18T20:23:12.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T20:23:12.999 INFO:journalctl@ceph.osd.8.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:23:13.201 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:23:13.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:13 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:13.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:13 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:13.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:13.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:13 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T20:23:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T20:23:14.496 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-18T20:23:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-18T20:23:14.496 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113428 s, 924 MB/s 2024-03-18T20:23:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-18T20:23:14.528 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:23:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:23:14.148+0000 7f3722f14640 -1 osd.8 0 log_to_monitors true 2024-03-18T20:23:14.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:14 smithi098 ceph-mon[43462]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:14.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:14 smithi098 ceph-mon[43462]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:14.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:14 smithi098 ceph-mon[43462]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:14.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:14 smithi169 ceph-mon[48529]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:14.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:14 smithi169 ceph-mon[48529]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:14.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:14 smithi169 ceph-mon[48529]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:14.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-18T20:23:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:14 smithi136 ceph-mon[48073]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:14 smithi136 ceph-mon[48073]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:14 smithi136 ceph-mon[48073]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T20:23:15.118 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:23:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T20:23:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: osdmap e54: 9 total, 8 up, 9 in 2024-03-18T20:23:15.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:15.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:15.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.884 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:23:15.533+0000 7f3718578700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T20:23:15.884 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:23:15.543+0000 7f3711b68700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: osdmap e54: 9 total, 8 up, 9 in 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:15.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T20:23:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: osdmap e54: 9 total, 8 up, 9 in 2024-03-18T20:23:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.076+0000 7f85f8bf1700 1 -- 172.21.15.98:0/3262816679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 msgr2=0x7f85f4103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.076+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/3262816679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4103c70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009990 tx=0x7f85e8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 -- 172.21.15.98:0/3262816679 shutdown_connections 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/3262816679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 0x7f85f4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/3262816679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4103c70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/3262816679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 0x7f85f41065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 -- 172.21.15.98:0/3262816679 >> 172.21.15.98:0/3262816679 conn(0x7f85f40fb2b0 msgr2=0x7f85f40fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 -- 172.21.15.98:0/3262816679 shutdown_connections 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 -- 172.21.15.98:0/3262816679 wait complete. 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 Processor -- start 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.077+0000 7f85f8bf1700 1 -- start start 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 0x7f85f41970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 0x7f85f419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f419e6d0 con 0x7f85f4101880 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f419e810 con 0x7f85f4106ae0 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f419e9a0 con 0x7f85f41041b0 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49136/0 (socket says 172.21.15.98:49136) 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 -- 172.21.15.98:0/2001152920 learned_addr learned my addr 172.21.15.98:0/2001152920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f2d9d700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 0x7f85f419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f1d9b700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 0x7f85f41970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 -- 172.21.15.98:0/2001152920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 msgr2=0x7f85f41970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 0x7f85f41970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 -- 172.21.15.98:0/2001152920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 msgr2=0x7f85f419e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 0x7f85f419e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 -- 172.21.15.98:0/2001152920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80095f0 con 0x7f85f4101880 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f259c700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85e8004730 tx=0x7f85e8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e801d070 con 0x7f85f4101880 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f419ec20 con 0x7f85f4101880 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85e800bdb0 con 0x7f85f4101880 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800fe90 con 0x7f85f4101880 2024-03-18T20:23:16.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.078+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f419f040 con 0x7f85f4101880 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.079+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f40fce30 con 0x7f85f4101880 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.081+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f85e800f660 con 0x7f85f4101880 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.081+0000 7f85e37fe700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 0x7f85dc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.081+0000 7f85f1d9b700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 0x7f85dc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.081+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5549+0+0 (secure 0 0 0) 0x7f85e80cd520 con 0x7f85f4101880 2024-03-18T20:23:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.082+0000 7f85f1d9b700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 0x7f85dc0a35e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009bf0 tx=0x7f85e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:16.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.085+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f85e8097810 con 0x7f85f4101880 2024-03-18T20:23:16.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:16.285+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f85f40611f0 con 0x7f85dc0a1120 2024-03-18T20:23:16.817 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi169 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi169 to 5704M 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: osdmap e55: 9 total, 8 up, 9 in 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] boot 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: osdmap e56: 9 total, 9 up, 9 in 2024-03-18T20:23:16.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi169 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi169 to 5704M 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: osdmap e55: 9 total, 8 up, 9 in 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:16.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] boot 2024-03-18T20:23:16.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: osdmap e56: 9 total, 9 up, 9 in 2024-03-18T20:23:16.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi169 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi169 to 5704M 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: osdmap e55: 9 total, 8 up, 9 in 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: osd.8 [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] boot 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: osdmap e56: 9 total, 9 up, 9 in 2024-03-18T20:23:16.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T20:23:17.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:17 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:23:17.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:17 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:23:17.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:17 smithi098 ceph-mon[43462]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:17.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:17 smithi098 ceph-mon[43462]: osdmap e57: 9 total, 9 up, 9 in 2024-03-18T20:23:17.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:17 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:23:17.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:17 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:23:17.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:17 smithi169 ceph-mon[48529]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:17.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:17 smithi169 ceph-mon[48529]: osdmap e57: 9 total, 9 up, 9 in 2024-03-18T20:23:17.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:17 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:23:17.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:17 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:23:17.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:17 smithi136 ceph-mon[48073]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:17.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:17 smithi136 ceph-mon[48073]: osdmap e57: 9 total, 9 up, 9 in 2024-03-18T20:23:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:18 smithi098 ceph-mon[43462]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:18 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]: dispatch 2024-03-18T20:23:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:18 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]': finished 2024-03-18T20:23:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:18 smithi098 ceph-mon[43462]: osdmap e58: 10 total, 9 up, 10 in 2024-03-18T20:23:18.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:18 smithi169 ceph-mon[48529]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:18 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]: dispatch 2024-03-18T20:23:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:18 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]': finished 2024-03-18T20:23:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:18 smithi169 ceph-mon[48529]: osdmap e58: 10 total, 9 up, 10 in 2024-03-18T20:23:18.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:18 smithi136 ceph-mon[48073]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:18 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]: dispatch 2024-03-18T20:23:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:18 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/457113972' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5969a1df-7157-4720-bb4e-d499f9a2d3d3"}]': finished 2024-03-18T20:23:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:18 smithi136 ceph-mon[48073]: osdmap e58: 10 total, 9 up, 10 in 2024-03-18T20:23:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:19.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:19 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1269833542' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:19 smithi098 ceph-mon[43462]: osdmap e59: 10 total, 9 up, 10 in 2024-03-18T20:23:19.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:19.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:19 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1269833542' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:19.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:19 smithi169 ceph-mon[48529]: osdmap e59: 10 total, 9 up, 10 in 2024-03-18T20:23:19.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:19 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1269833542' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:19 smithi136 ceph-mon[48073]: osdmap e59: 10 total, 9 up, 10 in 2024-03-18T20:23:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:20 smithi098 ceph-mon[43462]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:20 smithi098 ceph-mon[43462]: osdmap e60: 10 total, 9 up, 10 in 2024-03-18T20:23:20.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:20 smithi169 ceph-mon[48529]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:20 smithi169 ceph-mon[48529]: osdmap e60: 10 total, 9 up, 10 in 2024-03-18T20:23:20.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:20.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:20 smithi136 ceph-mon[48073]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:20.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:20 smithi136 ceph-mon[48073]: osdmap e60: 10 total, 9 up, 10 in 2024-03-18T20:23:20.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:22.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:22 smithi136 ceph-mon[48073]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T20:23:22.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:22 smithi098 ceph-mon[43462]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T20:23:22.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:22 smithi169 ceph-mon[48529]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T20:23:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:24 smithi169 ceph-mon[48529]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:23:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T20:23:24.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:24 smithi136 ceph-mon[48073]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:23:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T20:23:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:25.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:24 smithi098 ceph-mon[43462]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T20:23:25.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T20:23:25.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:25.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:25 smithi169 ceph-mon[48529]: Deploying daemon osd.9 on smithi098 2024-03-18T20:23:25.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:25 smithi098 ceph-mon[43462]: Deploying daemon osd.9 on smithi098 2024-03-18T20:23:25.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:25 smithi136 ceph-mon[48073]: Deploying daemon osd.9 on smithi098 2024-03-18T20:23:26.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:26 smithi169 ceph-mon[48529]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T20:23:26.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:26 smithi098 ceph-mon[43462]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T20:23:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:26 smithi136 ceph-mon[48073]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T20:23:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:27.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:27.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:27.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:27.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:27.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:27.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:28.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:28 smithi098 ceph-mon[43462]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T20:23:28.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:28 smithi169 ceph-mon[48529]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T20:23:28.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:28 smithi136 ceph-mon[48073]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T20:23:29.691 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-03-18T20:23:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.686+0000 7f85e37fe700 1 -- 172.21.15.98:0/2001152920 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f85f40611f0 con 0x7f85dc0a1120 2024-03-18T20:23:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 msgr2=0x7f85dc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 0x7f85dc0a35e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009bf0 tx=0x7f85e4009260 comp rx=0 tx=0).stop 2024-03-18T20:23:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 msgr2=0x7f85f4196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85e8004730 tx=0x7f85e8004760 comp rx=0 tx=0).stop 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 shutdown_connections 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f85dc0a1120 0x7f85dc0a35e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f4106ae0 0x7f85f419e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85f4101880 0x7f85f4196b90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 --2- 172.21.15.98:0/2001152920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85f41041b0 0x7f85f41970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.688+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 >> 172.21.15.98:0/2001152920 conn(0x7f85f40fb2b0 msgr2=0x7f85f4103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.689+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 shutdown_connections 2024-03-18T20:23:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:23:29.689+0000 7f85f8bf1700 1 -- 172.21.15.98:0/2001152920 wait complete. 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:29.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:29 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:29 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.074 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T20:23:30.076 INFO:tasks.cephadm:Deploying osd.10 on smithi136 with /dev/vg_nvme/lv_1... 2024-03-18T20:23:30.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T20:23:30.125 INFO:journalctl@ceph.osd.9.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:23:30.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:29 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:30.322 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:23:30.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:30 smithi098 ceph-mon[43462]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:23:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:30 smithi136 ceph-mon[48073]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:23:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:30 smithi169 ceph-mon[48529]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00723767 s, 1.4 GB/s 2024-03-18T20:23:31.690 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-18T20:23:31.747 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:23:31 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:23:31.455+0000 7fedf2024640 -1 osd.9 0 log_to_monitors true 2024-03-18T20:23:32.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-03-18T20:23:32.250 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:32.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:32.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:32.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:32.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:32.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:32.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:33.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 -- 172.21.15.136:0/3624137196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284102980 msgr2=0x7fe284109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 --2- 172.21.15.136:0/3624137196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284102980 0x7fe284109230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b0d0 tx=0x7fe27c00b3e0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 -- 172.21.15.136:0/3624137196 shutdown_connections 2024-03-18T20:23:33.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 --2- 172.21.15.136:0/3624137196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284102980 0x7fe284109230 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 --2- 172.21.15.136:0/3624137196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 0x7fe284101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 --2- 172.21.15.136:0/3624137196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284101fe0 0x7fe284102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 -- 172.21.15.136:0/3624137196 >> 172.21.15.136:0/3624137196 conn(0x7fe2840fc380 msgr2=0x7fe2840fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:33.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 -- 172.21.15.136:0/3624137196 shutdown_connections 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.288+0000 7fe288d70700 1 -- 172.21.15.136:0/3624137196 wait complete. 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 Processor -- start 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 -- start start 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 0x7fe284074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284102980 0x7fe2840736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284073e80 con 0x7fe284100de0 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284073ff0 con 0x7fe284101fe0 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe288d70700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841a0d80 con 0x7fe284102980 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49726/0 (socket says 172.21.15.136:49726) 2024-03-18T20:23:33.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 -- 172.21.15.136:0/228773726 learned_addr learned my addr 172.21.15.136:0/228773726 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe28259c700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 0x7fe284074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe282d9d700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284102980 0x7fe2840736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 -- 172.21.15.136:0/228773726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284102980 msgr2=0x7fe2840736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284102980 0x7fe2840736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 -- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 msgr2=0x7fe284074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 0x7fe284074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.289+0000 7fe281d9b700 1 -- 172.21.15.136:0/228773726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c009d00 con 0x7fe284101fe0 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe281d9b700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe27800d950 tx=0x7fe27800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278009940 con 0x7fe284101fe0 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe278010460 con 0x7fe284101fe0 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27800f620 con 0x7fe284101fe0 2024-03-18T20:23:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2841a0fb0 con 0x7fe284101fe0 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.290+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2841a13d0 con 0x7fe284101fe0 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.291+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe284108210 con 0x7fe284101fe0 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.292+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe2780105d0 con 0x7fe284101fe0 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.292+0000 7fe2737fe700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 0x7fe26c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.293+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fe2780cbde0 con 0x7fe284101fe0 2024-03-18T20:23:33.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.293+0000 7fe28259c700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 0x7fe26c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.293+0000 7fe28259c700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 0x7fe26c0a35e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe284101e40 tx=0x7fe27400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:33.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.297+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe278095ea0 con 0x7fe284101fe0 2024-03-18T20:23:33.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:33.502+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2840611f0 con 0x7fe26c0a1120 2024-03-18T20:23:33.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi098 2024-03-18T20:23:33.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi098 to 4278M 2024-03-18T20:23:33.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T20:23:33.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: osdmap e61: 10 total, 9 up, 10 in 2024-03-18T20:23:33.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:33.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:33 smithi136 ceph-mon[48073]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:23:33.843 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:23:33.413+0000 7fede7688700 -1 osd.9 0 waiting for initial osdmap 2024-03-18T20:23:33.844 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:23:33.431+0000 7fede347d700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi098 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi098 to 4278M 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: osdmap e61: 10 total, 9 up, 10 in 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:33.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:33 smithi098 ceph-mon[43462]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:23:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi098 2024-03-18T20:23:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi098 to 4278M 2024-03-18T20:23:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T20:23:33.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: osdmap e61: 10 total, 9 up, 10 in 2024-03-18T20:23:33.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:33.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:33 smithi169 ceph-mon[48529]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-18T20:23:34.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:23:34.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: osdmap e62: 10 total, 9 up, 10 in 2024-03-18T20:23:34.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:34.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:34.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:34.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:34.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:34 smithi136 ceph-mon[48073]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: osdmap e62: 10 total, 9 up, 10 in 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:34.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:34.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:34 smithi098 ceph-mon[43462]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-18T20:23:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: osdmap e62: 10 total, 9 up, 10 in 2024-03-18T20:23:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:34.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:34 smithi169 ceph-mon[48529]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-18T20:23:35.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] boot 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: osdmap e63: 10 total, 10 up, 10 in 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/800394319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]': finished 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: osdmap e64: 11 total, 10 up, 11 in 2024-03-18T20:23:35.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] boot 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: osdmap e63: 10 total, 10 up, 10 in 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/800394319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]': finished 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: osdmap e64: 11 total, 10 up, 11 in 2024-03-18T20:23:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: osd.9 [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] boot 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: osdmap e63: 10 total, 10 up, 10 in 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/800394319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]: dispatch 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbb5b70e-d702-48c4-9811-d1d75e09de82"}]': finished 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: osdmap e64: 11 total, 10 up, 11 in 2024-03-18T20:23:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:36.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:36 smithi136 ceph-mon[48073]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:36.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:36 smithi136 ceph-mon[48073]: from='client.? 172.21.15.136:0/3455165754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:36.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:36 smithi136 ceph-mon[48073]: osdmap e65: 11 total, 10 up, 11 in 2024-03-18T20:23:36.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:36 smithi098 ceph-mon[43462]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:36 smithi098 ceph-mon[43462]: from='client.? 172.21.15.136:0/3455165754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:36 smithi098 ceph-mon[43462]: osdmap e65: 11 total, 10 up, 11 in 2024-03-18T20:23:36.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:36.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:36 smithi169 ceph-mon[48529]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:36.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:36 smithi169 ceph-mon[48529]: from='client.? 172.21.15.136:0/3455165754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:36.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:36 smithi169 ceph-mon[48529]: osdmap e65: 11 total, 10 up, 11 in 2024-03-18T20:23:36.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:38.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:38 smithi169 ceph-mon[48529]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:38.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:38 smithi136 ceph-mon[48073]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:38 smithi098 ceph-mon[43462]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:40.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:40 smithi136 ceph-mon[48073]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:40.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:40 smithi169 ceph-mon[48529]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:40 smithi098 ceph-mon[43462]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:41.820 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T20:23:41.820 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:41.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T20:23:41.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T20:23:42.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:42.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:42 smithi136 ceph-mon[48073]: Deploying daemon osd.10 on smithi136 2024-03-18T20:23:42.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:42 smithi136 ceph-mon[48073]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:42 smithi169 ceph-mon[48529]: Deploying daemon osd.10 on smithi136 2024-03-18T20:23:42.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:42 smithi169 ceph-mon[48529]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:43.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:42 smithi098 ceph-mon[43462]: Deploying daemon osd.10 on smithi136 2024-03-18T20:23:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:42 smithi098 ceph-mon[43462]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:44 smithi169 ceph-mon[48529]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:44 smithi136 ceph-mon[48073]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:44 smithi098 ceph-mon[43462]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:45.822 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:45.823 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:45.823 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:46.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 10 on host 'smithi136' 2024-03-18T20:23:46.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.855+0000 7fe2737fe700 1 -- 172.21.15.136:0/228773726 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe2840611f0 con 0x7fe26c0a1120 2024-03-18T20:23:46.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.857+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 msgr2=0x7fe26c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:46.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.857+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 0x7fe26c0a35e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe284101e40 tx=0x7fe27400b410 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.857+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 msgr2=0x7fe2840731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.857+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe27800d950 tx=0x7fe27800dc60 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 shutdown_connections 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe284102980 0x7fe2840736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe284100de0 0x7fe284074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe26c0a1120 0x7fe26c0a35e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 --2- 172.21.15.136:0/228773726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe284101fe0 0x7fe2840731a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 >> 172.21.15.136:0/228773726 conn(0x7fe2840fc380 msgr2=0x7fe284105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 shutdown_connections 2024-03-18T20:23:46.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-18T20:23:46.858+0000 7fe288d70700 1 -- 172.21.15.136:0/228773726 wait complete. 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.236 DEBUG:teuthology.orchestra.run.smithi136:osd.10> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T20:23:47.238 INFO:tasks.cephadm:Deploying osd.11 on smithi169 with /dev/vg_nvme/lv_1... 2024-03-18T20:23:47.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T20:23:47.271 INFO:journalctl@ceph.osd.10.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:23:47.481 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:23:47.839 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:47.840 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.840 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:47.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:47.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:23:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:48.853 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T20:23:48.853 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T20:23:48.854 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-18T20:23:48.854 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-18T20:23:48.854 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131875 s, 795 MB/s 2024-03-18T20:23:48.854 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-18T20:23:48.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:23:48 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:23:48.709+0000 7fa76d7e8640 -1 osd.10 0 log_to_monitors true 2024-03-18T20:23:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:48 smithi136 ceph-mon[48073]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:48 smithi136 ceph-mon[48073]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:48.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:48 smithi136 ceph-mon[48073]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:48 smithi098 ceph-mon[43462]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:48 smithi098 ceph-mon[43462]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:49.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:48 smithi098 ceph-mon[43462]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:48 smithi169 ceph-mon[48529]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:48 smithi169 ceph-mon[48529]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:48 smithi169 ceph-mon[48529]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T20:23:49.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2024-03-18T20:23:49.488 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:23:49.976 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: osdmap e66: 11 total, 10 up, 11 in 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:49.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:49.978 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:49.978 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:49.978 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:49.978 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:49 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.241 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:23:49.844+0000 7fa762e4c700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T20:23:50.241 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:23:49.858+0000 7fa75ac39700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:23:50.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T20:23:50.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: osdmap e66: 11 total, 10 up, 11 in 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:50.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:50.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:49 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: osdmap e66: 11 total, 10 up, 11 in 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:23:50.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:50.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:23:50.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:49 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:23:50.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.543+0000 7fed64a67700 1 -- 172.21.15.169:0/1185357277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed60100f30 msgr2=0x7fed601013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.543+0000 7fed64a67700 1 --2- 172.21.15.169:0/1185357277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed60100f30 0x7fed601013b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed54009990 tx=0x7fed54009ca0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 -- 172.21.15.169:0/1185357277 shutdown_connections 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 --2- 172.21.15.169:0/1185357277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed601018f0 0x7fed60108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 --2- 172.21.15.169:0/1185357277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100210 0x7fed60100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 --2- 172.21.15.169:0/1185357277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed60100f30 0x7fed601013b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 -- 172.21.15.169:0/1185357277 >> 172.21.15.169:0/1185357277 conn(0x7fed600faa10 msgr2=0x7fed600fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 -- 172.21.15.169:0/1185357277 shutdown_connections 2024-03-18T20:23:50.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 -- 172.21.15.169:0/1185357277 wait complete. 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.544+0000 7fed64a67700 1 Processor -- start 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 -- start start 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed60100210 0x7fed60103e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100f30 0x7fed601024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed60103180 con 0x7fed60100f30 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed601032f0 con 0x7fed60100210 2024-03-18T20:23:50.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed601a0c90 con 0x7fed601018f0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51554/0 (socket says 172.21.15.169:51554) 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 -- 172.21.15.169:0/2047114190 learned_addr learned my addr 172.21.15.169:0/2047114190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5e59c700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed60100210 0x7fed60103e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5dd9b700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100f30 0x7fed601024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 -- 172.21.15.169:0/2047114190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed60100210 msgr2=0x7fed60103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed60100210 0x7fed60103e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 -- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100f30 msgr2=0x7fed601024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100f30 0x7fed601024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 -- 172.21.15.169:0/2047114190 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540095f0 con 0x7fed601018f0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed5ed9d700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5800b970 tx=0x7fed5800bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5800c9b0 con 0x7fed601018f0 2024-03-18T20:23:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed601a0f20 con 0x7fed601018f0 2024-03-18T20:23:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed58014440 con 0x7fed601018f0 2024-03-18T20:23:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.545+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed580075d0 con 0x7fed601018f0 2024-03-18T20:23:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.546+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed601a1390 con 0x7fed601018f0 2024-03-18T20:23:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.546+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6004ea80 con 0x7fed601018f0 2024-03-18T20:23:50.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.548+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fed580145b0 con 0x7fed601018f0 2024-03-18T20:23:50.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.548+0000 7fed4f7fe700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 0x7fed480a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.548+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fed58006e40 con 0x7fed601018f0 2024-03-18T20:23:50.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.548+0000 7fed5e59c700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 0x7fed480a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.549+0000 7fed5e59c700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 0x7fed480a35e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed50005fd0 tx=0x7fed50005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:50.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.552+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed58096090 con 0x7fed601018f0 2024-03-18T20:23:50.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:23:50.752+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fed600ff480 con 0x7fed480a1120 2024-03-18T20:23:51.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi136 2024-03-18T20:23:51.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi136 to 4278M 2024-03-18T20:23:51.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:51.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: osdmap e67: 11 total, 10 up, 11 in 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:51.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:50 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:51.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi136 2024-03-18T20:23:51.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi136 to 4278M 2024-03-18T20:23:51.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:51.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: osdmap e67: 11 total, 10 up, 11 in 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:51.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:50 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:51.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi136 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi136 to 4278M 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: osdmap e67: 11 total, 10 up, 11 in 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T20:23:51.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:50 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:52.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] boot 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: osdmap e68: 11 total, 11 up, 11 in 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:52.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:51 smithi169 ceph-mon[48529]: osdmap e69: 11 total, 11 up, 11 in 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] boot 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: osdmap e68: 11 total, 11 up, 11 in 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:52.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:51 smithi136 ceph-mon[48073]: osdmap e69: 11 total, 11 up, 11 in 2024-03-18T20:23:52.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: osd.10 [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] boot 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: osdmap e68: 11 total, 11 up, 11 in 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T20:23:52.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:51 smithi098 ceph-mon[43462]: osdmap e69: 11 total, 11 up, 11 in 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: osdmap e70: 11 total, 11 up, 11 in 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/2995210764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]': finished 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: osdmap e71: 12 total, 11 up, 12 in 2024-03-18T20:23:53.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:52 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: osdmap e70: 11 total, 11 up, 11 in 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/2995210764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]': finished 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: osdmap e71: 12 total, 11 up, 12 in 2024-03-18T20:23:53.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:52 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:53.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: osdmap e70: 11 total, 11 up, 11 in 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/2995210764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]: dispatch 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50289373-3457-45bd-8c72-de8ae15b6ec2"}]': finished 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: osdmap e71: 12 total, 11 up, 12 in 2024-03-18T20:23:53.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:52 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:54.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:53 smithi169 ceph-mon[48529]: from='client.? 172.21.15.169:0/3890106517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:53 smithi169 ceph-mon[48529]: osdmap e72: 12 total, 11 up, 12 in 2024-03-18T20:23:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:53 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:54.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:53 smithi136 ceph-mon[48073]: from='client.? 172.21.15.169:0/3890106517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:54.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:53 smithi136 ceph-mon[48073]: osdmap e72: 12 total, 11 up, 12 in 2024-03-18T20:23:54.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:53 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:54.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:53 smithi098 ceph-mon[43462]: from='client.? 172.21.15.169:0/3890106517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T20:23:54.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:53 smithi098 ceph-mon[43462]: osdmap e72: 12 total, 11 up, 12 in 2024-03-18T20:23:54.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:53 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:54 smithi169 ceph-mon[48529]: pgmap v185: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:54 smithi169 ceph-mon[48529]: osdmap e73: 12 total, 11 up, 12 in 2024-03-18T20:23:55.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:54 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:54 smithi136 ceph-mon[48073]: pgmap v185: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:54 smithi136 ceph-mon[48073]: osdmap e73: 12 total, 11 up, 12 in 2024-03-18T20:23:55.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:54 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:55.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:54 smithi098 ceph-mon[43462]: pgmap v185: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:23:55.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:54 smithi098 ceph-mon[43462]: osdmap e73: 12 total, 11 up, 12 in 2024-03-18T20:23:55.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:54 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:23:57.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:56 smithi169 ceph-mon[48529]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T20:23:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:56 smithi136 ceph-mon[48073]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T20:23:57.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:56 smithi098 ceph-mon[43462]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T20:23:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:58 smithi169 ceph-mon[48529]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T20:23:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T20:23:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:58 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:59.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:58 smithi136 ceph-mon[48073]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T20:23:59.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T20:23:59.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:58 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:59.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:58 smithi098 ceph-mon[43462]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-18T20:23:59.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T20:23:59.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:58 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:23:59.998 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:23:59 smithi169 ceph-mon[48529]: Deploying daemon osd.11 on smithi169 2024-03-18T20:24:00.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:23:59 smithi136 ceph-mon[48073]: Deploying daemon osd.11 on smithi169 2024-03-18T20:24:00.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:23:59 smithi098 ceph-mon[43462]: Deploying daemon osd.11 on smithi169 2024-03-18T20:24:00.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:00 smithi169 ceph-mon[48529]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T20:24:01.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:00 smithi136 ceph-mon[48073]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T20:24:01.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:00 smithi098 ceph-mon[43462]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T20:24:02.633 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:02 smithi169 ceph-mon[48529]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:24:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:02.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:02 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:02 smithi136 ceph-mon[48073]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:24:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:02 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:02 smithi098 ceph-mon[43462]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T20:24:02.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:02.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:02.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:02 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.559 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 11 on host 'smithi169' 2024-03-18T20:24:04.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.555+0000 7fed4f7fe700 1 -- 172.21.15.169:0/2047114190 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fed600ff480 con 0x7fed480a1120 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.557+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 msgr2=0x7fed480a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.557+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 0x7fed480a35e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fed50005fd0 tx=0x7fed50005e50 comp rx=0 tx=0).stop 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 msgr2=0x7fed601029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5800b970 tx=0x7fed5800bd30 comp rx=0 tx=0).stop 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 shutdown_connections 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed601018f0 0x7fed601029e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fed480a1120 0x7fed480a35e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:04.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed60100210 0x7fed60103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:04.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 --2- 172.21.15.169:0/2047114190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed60100f30 0x7fed601024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:04.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 >> 172.21.15.169:0/2047114190 conn(0x7fed600faa10 msgr2=0x7fed600fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:04.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 shutdown_connections 2024-03-18T20:24:04.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:04.558+0000 7fed64a67700 1 -- 172.21.15.169:0/2047114190 wait complete. 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:04.841 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:04 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:04.959 DEBUG:teuthology.orchestra.run.smithi169:osd.11> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T20:24:04.961 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T20:24:04.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:04 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:04.992 INFO:journalctl@ceph.osd.11.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:24:04.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T20:24:04.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:04.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:04.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:04.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:04 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:05.199 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:24:06.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1284094184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 msgr2=0x7f3ac81024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:06.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1284094184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac81024f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009990 tx=0x7f3ac4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1284094184 shutdown_connections 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1284094184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 0x7f3ac81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1284094184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac80ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1284094184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac81024f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1284094184 >> 172.21.15.98:0/1284094184 conn(0x7f3ac80faa10 msgr2=0x7f3ac80fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1284094184 shutdown_connections 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.156+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1284094184 wait complete. 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 Processor -- start 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 -- start start 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac81019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 0x7f3ac8100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8100d20 con 0x7f3ac8102090 2024-03-18T20:24:06.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8100e90 con 0x7f3ac80ff0b0 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ad02e1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81a0da0 con 0x7f3ac8102a30 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35884/0 (socket says 172.21.15.98:35884) 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 -- 172.21.15.98:0/1733489802 learned_addr learned my addr 172.21.15.98:0/1733489802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ace87e700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 0x7f3ac8100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ace07d700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac81019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 -- 172.21.15.98:0/1733489802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 msgr2=0x7f3ac8100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 0x7f3ac8100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 -- 172.21.15.98:0/1733489802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 msgr2=0x7f3ac81019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac81019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3acd87c700 1 -- 172.21.15.98:0/1733489802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40095f0 con 0x7f3ac8102090 2024-03-18T20:24:06.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.157+0000 7f3ace07d700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3acd87c700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40098e0 tx=0x7f3ac4003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac401d070 con 0x7f3ac8102090 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3ac400bd90 con 0x7f3ac8102090 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac81a0fd0 con 0x7f3ac8102090 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400f460 con 0x7f3ac8102090 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.158+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac81a1460 con 0x7f3ac8102090 2024-03-18T20:24:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.159+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac8108360 con 0x7f3ac8102090 2024-03-18T20:24:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.160+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f3ac4022470 con 0x7f3ac8102090 2024-03-18T20:24:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.160+0000 7f3abf7fe700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 0x7f3ab40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.161+0000 7f3ace07d700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 0x7f3ab40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.161+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3ac40cdb70 con 0x7f3ac8102090 2024-03-18T20:24:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.161+0000 7f3ace07d700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 0x7f3ab40a3660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8068550 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:06.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.164+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ac4097970 con 0x7f3ac8102090 2024-03-18T20:24:06.343 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:24:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:24:06.336+0000 7fb2db077640 -1 osd.11 0 log_to_monitors true 2024-03-18T20:24:06.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.360+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3ac804ea80 con 0x7f3ac8102090 2024-03-18T20:24:06.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.360+0000 7f3abf7fe700 1 -- 172.21.15.98:0/1733489802 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) v1 ==== 74+0+133 (secure 0 0 0) 0x7f3ac409af90 con 0x7f3ac8102090 2024-03-18T20:24:06.365 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 msgr2=0x7f3ab40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 0x7f3ab40a3660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8068550 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 msgr2=0x7f3ac8100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40098e0 tx=0x7f3ac4003a50 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 shutdown_connections 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ac8102a30 0x7f3ac8100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3ab40a11a0 0x7f3ab40a3660 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ac80ff0b0 0x7f3ac81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 --2- 172.21.15.98:0/1733489802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ac8102090 0x7f3ac8100040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 >> 172.21.15.98:0/1733489802 conn(0x7f3ac80faa10 msgr2=0x7f3ac81054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 shutdown_connections 2024-03-18T20:24:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:06.362+0000 7f3ad02e1700 1 -- 172.21.15.98:0/1733489802 wait complete. 2024-03-18T20:24:06.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:06 smithi098 ceph-mon[43462]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-18T20:24:06.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:06 smithi098 ceph-mon[43462]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T20:24:06.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:06 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1733489802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:06.724 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1710793430,"num_in_osds":12,"osd_in_since":1710793432,"num_remapped_pgs":0} 2024-03-18T20:24:06.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:06 smithi169 ceph-mon[48529]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-18T20:24:06.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:06 smithi169 ceph-mon[48529]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T20:24:06.941 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:06 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1733489802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:06 smithi136 ceph-mon[48073]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-18T20:24:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:06 smithi136 ceph-mon[48073]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T20:24:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:06 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1733489802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:07.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T20:24:07.884 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:24:07.626+0000 7fb2d06db700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T20:24:07.884 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:24:07.646+0000 7fb2c8cc9700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T20:24:07.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: osdmap e74: 12 total, 11 up, 12 in 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.885 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:24:07.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: osdmap e75: 12 total, 11 up, 12 in 2024-03-18T20:24:07.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:07 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:07.954 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: osdmap e74: 12 total, 11 up, 12 in 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:07.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: osdmap e75: 12 total, 11 up, 12 in 2024-03-18T20:24:07.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:07 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:08.052 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T20:24:08.052 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: osdmap e74: 12 total, 11 up, 12 in 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:08.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-18T20:24:08.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: osdmap e75: 12 total, 11 up, 12 in 2024-03-18T20:24:08.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:07 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- 172.21.15.98:0/2443256445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 msgr2=0x7fb50c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 --2- 172.21.15.98:0/2443256445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c106600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009990 tx=0x7fb4fc009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- 172.21.15.98:0/2443256445 shutdown_connections 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 --2- 172.21.15.98:0/2443256445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c108fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 --2- 172.21.15.98:0/2443256445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 0x7fb50c103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 --2- 172.21.15.98:0/2443256445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c106600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- 172.21.15.98:0/2443256445 >> 172.21.15.98:0/2443256445 conn(0x7fb50c0fb310 msgr2=0x7fb50c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- 172.21.15.98:0/2443256445 shutdown_connections 2024-03-18T20:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- 172.21.15.98:0/2443256445 wait complete. 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 Processor -- start 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.864+0000 7fb511535700 1 -- start start 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 0x7fb50c196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19e770 con 0x7fb50c104210 2024-03-18T20:24:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19e8b0 con 0x7fb50c106b40 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19ea70 con 0x7fb50c1018e0 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35902/0 (socket says 172.21.15.98:35902) 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 -- 172.21.15.98:0/1708719565 learned_addr learned my addr 172.21.15.98:0/1708719565 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50affd700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 0x7fb50c196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50b7fe700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 -- 172.21.15.98:0/1708719565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 msgr2=0x7fb50c196c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 0x7fb50c196c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 -- 172.21.15.98:0/1708719565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 msgr2=0x7fb50c19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 -- 172.21.15.98:0/1708719565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0095f0 con 0x7fb50c104210 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50b7fe700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb50a7fc700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098e0 tx=0x7fb4fc003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc01d070 con 0x7fb50c104210 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50c19ecf0 con 0x7fb50c104210 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb4fc00bd90 con 0x7fb50c104210 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc022d20 con 0x7fb50c104210 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.865+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50c19f160 con 0x7fb50c104210 2024-03-18T20:24:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.866+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50c0fce90 con 0x7fb50c104210 2024-03-18T20:24:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.868+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb4fc003f40 con 0x7fb50c104210 2024-03-18T20:24:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.868+0000 7fb4f3fff700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 0x7fb4f40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.868+0000 7fb50affd700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 0x7fb4f40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.868+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb4fc0ce050 con 0x7fb50c104210 2024-03-18T20:24:08.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.869+0000 7fb50affd700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 0x7fb4f40a35e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb50c198100 tx=0x7fb50000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:08.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:08.872+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb4fc097d10 con 0x7fb50c104210 2024-03-18T20:24:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:08 smithi136 ceph-mon[48073]: Detected new or changed devices on smithi169 2024-03-18T20:24:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:08 smithi136 ceph-mon[48073]: Adjusting osd_memory_target on smithi169 to 4278M 2024-03-18T20:24:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:08 smithi136 ceph-mon[48073]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:08 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.065+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb50c02ccb0 con 0x7fb50c104210 2024-03-18T20:24:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.066+0000 7fb4f3fff700 1 -- 172.21.15.98:0/1708719565 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fb4fc09b330 con 0x7fb50c104210 2024-03-18T20:24:09.071 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 msgr2=0x7fb4f40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 0x7fb4f40a35e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb50c198100 tx=0x7fb50000b410 comp rx=0 tx=0).stop 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 msgr2=0x7fb50c1971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098e0 tx=0x7fb4fc003a50 comp rx=0 tx=0).stop 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 shutdown_connections 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb4f40a1120 0x7fb4f40a35e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c106b40 0x7fb50c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c1018e0 0x7fb50c196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 --2- 172.21.15.98:0/1708719565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb50c104210 0x7fb50c1971b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 >> 172.21.15.98:0/1708719565 conn(0x7fb50c0fb310 msgr2=0x7fb50c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 shutdown_connections 2024-03-18T20:24:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:09.068+0000 7fb511535700 1 -- 172.21.15.98:0/1708719565 wait complete. 2024-03-18T20:24:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:08 smithi098 ceph-mon[43462]: Detected new or changed devices on smithi169 2024-03-18T20:24:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:08 smithi098 ceph-mon[43462]: Adjusting osd_memory_target on smithi169 to 4278M 2024-03-18T20:24:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:08 smithi098 ceph-mon[43462]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:08 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:08 smithi169 ceph-mon[48529]: Detected new or changed devices on smithi169 2024-03-18T20:24:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:08 smithi169 ceph-mon[48529]: Adjusting osd_memory_target on smithi169 to 4278M 2024-03-18T20:24:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:08 smithi169 ceph-mon[48529]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:08 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.440 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1710793448,"num_in_osds":12,"osd_in_since":1710793432,"num_remapped_pgs":0} 2024-03-18T20:24:09.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T20:24:09.661 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:24:09.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: purged_snaps scrub starts 2024-03-18T20:24:09.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: purged_snaps scrub ok 2024-03-18T20:24:09.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] boot 2024-03-18T20:24:09.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: osdmap e76: 12 total, 12 up, 12 in 2024-03-18T20:24:09.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:09 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1708719565' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: purged_snaps scrub starts 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: purged_snaps scrub ok 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] boot 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: osdmap e76: 12 total, 12 up, 12 in 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:09 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1708719565' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: purged_snaps scrub starts 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: purged_snaps scrub ok 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: osd.11 [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] boot 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: osdmap e76: 12 total, 12 up, 12 in 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T20:24:10.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:09 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1708719565' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 -- 172.21.15.98:0/941984454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 msgr2=0x7f67cc1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 --2- 172.21.15.98:0/941984454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc1072d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f67c00099c0 tx=0x7f67c0009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 -- 172.21.15.98:0/941984454 shutdown_connections 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 --2- 172.21.15.98:0/941984454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc101820 0x7f67cc10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 --2- 172.21.15.98:0/941984454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc1072d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 --2- 172.21.15.98:0/941984454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc100e60 0x7f67cc1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 -- 172.21.15.98:0/941984454 >> 172.21.15.98:0/941984454 conn(0x7f67cc077fa0 msgr2=0x7f67cc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 -- 172.21.15.98:0/941984454 shutdown_connections 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.581+0000 7f67d2d25700 1 -- 172.21.15.98:0/941984454 wait complete. 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 Processor -- start 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 -- start start 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc100e60 0x7f67cc1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 0x7f67cc1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc19c060 con 0x7f67cc106ef0 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc19c1d0 con 0x7f67cc100e60 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d2d25700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc19c390 con 0x7f67cc101820 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35920/0 (socket says 172.21.15.98:35920) 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 -- 172.21.15.98:0/668574445 learned_addr learned my addr 172.21.15.98:0/668574445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:10.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67cbfff700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 0x7f67cc1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d0ac1700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc100e60 0x7f67cc1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 -- 172.21.15.98:0/668574445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 msgr2=0x7f67cc1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 0x7f67cc1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 -- 172.21.15.98:0/668574445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc100e60 msgr2=0x7f67cc1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc100e60 0x7f67cc1971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67d12c2700 1 -- 172.21.15.98:0/668574445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00095f0 con 0x7f67cc106ef0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.582+0000 7f67cbfff700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 0x7f67cc1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67d12c2700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f67c400b9e0 tx=0x7f67c400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c400c950 con 0x7f67cc106ef0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f67c4014440 con 0x7f67cc106ef0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc19c670 con 0x7f67cc106ef0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c40075d0 con 0x7f67cc106ef0 2024-03-18T20:24:10.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.583+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc1a3ed0 con 0x7f67cc106ef0 2024-03-18T20:24:10.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.584+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc1092d0 con 0x7f67cc106ef0 2024-03-18T20:24:10.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.585+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f67c40145b0 con 0x7f67cc106ef0 2024-03-18T20:24:10.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.586+0000 7f67c9ffb700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 0x7f67bc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:10.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.586+0000 7f67d0ac1700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 0x7f67bc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:10.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.586+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f67c4006e40 con 0x7f67cc106ef0 2024-03-18T20:24:10.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.586+0000 7f67d0ac1700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 0x7f67bc0a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f67c00036c0 tx=0x7f67c0009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:10.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.590+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f67c40958d0 con 0x7f67cc106ef0 2024-03-18T20:24:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.783+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67cc04ea80 con 0x7f67cc106ef0 2024-03-18T20:24:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.784+0000 7f67c9ffb700 1 -- 172.21.15.98:0/668574445 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18422 (secure 0 0 0) 0x7f67c4098ef0 con 0x7f67cc106ef0 2024-03-18T20:24:10.789 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:10.790 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":77,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:09.636901+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 msgr2=0x7f67bc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 0x7f67bc0a3660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f67c00036c0 tx=0x7f67c0009f10 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 msgr2=0x7f67cc19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f67c400b9e0 tx=0x7f67c400bda0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 shutdown_connections 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67cc106ef0 0x7f67cc19ba50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67bc0a11a0 0x7f67bc0a3660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc100e60 0x7f67cc1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 --2- 172.21.15.98:0/668574445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67cc101820 0x7f67cc1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 >> 172.21.15.98:0/668574445 conn(0x7f67cc077fa0 msgr2=0x7f67cc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 shutdown_connections 2024-03-18T20:24:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:10.786+0000 7f67d2d25700 1 -- 172.21.15.98:0/668574445 wait complete. 2024-03-18T20:24:10.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:10 smithi098 ceph-mon[43462]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:10.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:10 smithi098 ceph-mon[43462]: osdmap e77: 12 total, 12 up, 12 in 2024-03-18T20:24:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:10 smithi136 ceph-mon[48073]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:10 smithi136 ceph-mon[48073]: osdmap e77: 12 total, 12 up, 12 in 2024-03-18T20:24:11.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:10 smithi169 ceph-mon[48529]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-03-18T20:24:11.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:10 smithi169 ceph-mon[48529]: osdmap e77: 12 total, 12 up, 12 in 2024-03-18T20:24:11.148 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:24:11.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T20:24:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:24:11.810 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:11 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/668574445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:11.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:11 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/668574445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:12.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:11 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/668574445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- 172.21.15.98:0/219941601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d98106df0 msgr2=0x7f4d981071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/219941601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d98106df0 0x7f4d981071d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4d940099c0 tx=0x7f4d94009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- 172.21.15.98:0/219941601 shutdown_connections 2024-03-18T20:24:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/219941601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d980699d0 0x7f4d9810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/219941601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d98106df0 0x7f4d981071d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/219941601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98069490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- 172.21.15.98:0/219941601 >> 172.21.15.98:0/219941601 conn(0x7f4d98076c80 msgr2=0x7f4d98077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- 172.21.15.98:0/219941601 shutdown_connections 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- 172.21.15.98:0/219941601 wait complete. 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 Processor -- start 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 -- start start 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.335+0000 7f4d9f72f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d98106df0 0x7f4d9819b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9819bf30 con 0x7f4d980699d0 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9819c0a0 con 0x7f4d98106df0 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9819c260 con 0x7f4d98069010 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35926/0 (socket says 172.21.15.98:35926) 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 -- 172.21.15.98:0/3766824851 learned_addr learned my addr 172.21.15.98:0/3766824851 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9dccc700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d98106df0 0x7f4d9819b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9d4cb700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 -- 172.21.15.98:0/3766824851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 msgr2=0x7f4d98197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 -- 172.21.15.98:0/3766824851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d98106df0 msgr2=0x7f4d9819b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d98106df0 0x7f4d9819b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 -- 172.21.15.98:0/3766824851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d940095f0 con 0x7f4d980699d0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9d4cb700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9ccca700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800d980 tx=0x7f4d8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88004220 con 0x7f4d980699d0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4d88007640 con 0x7f4d980699d0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880055e0 con 0x7f4d980699d0 2024-03-18T20:24:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9819c540 con 0x7f4d980699d0 2024-03-18T20:24:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.336+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d981a3da0 con 0x7f4d980699d0 2024-03-18T20:24:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.337+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d981092f0 con 0x7f4d980699d0 2024-03-18T20:24:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.338+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4d88004750 con 0x7f4d980699d0 2024-03-18T20:24:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.339+0000 7f4d8e7fc700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 0x7f4d840a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.339+0000 7f4d9d4cb700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 0x7f4d840a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.339+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4d880cc720 con 0x7f4d980699d0 2024-03-18T20:24:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.339+0000 7f4d9d4cb700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 0x7f4d840a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d94006010 tx=0x7f4d94003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.342+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d88096350 con 0x7f4d980699d0 2024-03-18T20:24:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.535+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d98198340 con 0x7f4d980699d0 2024-03-18T20:24:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.535+0000 7f4d8e7fc700 1 -- 172.21.15.98:0/3766824851 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d880160a0 con 0x7f4d980699d0 2024-03-18T20:24:12.540 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 msgr2=0x7f4d840a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 0x7f4d840a3660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d94006010 tx=0x7f4d94003680 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 msgr2=0x7f4d98197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800d980 tx=0x7f4d8800dc90 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 shutdown_connections 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d840a11a0 0x7f4d840a3660 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d98106df0 0x7f4d9819b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d98069010 0x7f4d98197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 --2- 172.21.15.98:0/3766824851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d980699d0 0x7f4d98197570 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 >> 172.21.15.98:0/3766824851 conn(0x7f4d98076c80 msgr2=0x7f4d980feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 shutdown_connections 2024-03-18T20:24:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:12.537+0000 7f4d9f72f700 1 -- 172.21.15.98:0/3766824851 wait complete. 2024-03-18T20:24:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:12 smithi098 ceph-mon[43462]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:12 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3766824851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:12.890 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-03-18T20:24:12.890 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-03-18T20:24:12.890 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.b on smithi136 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.e on smithi136 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.h on smithi136 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.k on smithi136 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.c on smithi169 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.f on smithi169 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.i on smithi169 2024-03-18T20:24:12.891 INFO:tasks.cephadm:Adding mds.l on smithi169 2024-03-18T20:24:12.891 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l' 2024-03-18T20:24:12.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:12 smithi169 ceph-mon[48529]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:12.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:12 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3766824851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:12.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:12 smithi136 ceph-mon[48073]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:12.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:12 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3766824851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:13.127 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:24:14.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 -- 172.21.15.169:0/4052018815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 msgr2=0x7fb6541093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:14.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 --2- 172.21.15.169:0/4052018815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb6541093a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b0a0 tx=0x7fb64c00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 -- 172.21.15.169:0/4052018815 shutdown_connections 2024-03-18T20:24:14.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 --2- 172.21.15.169:0/4052018815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb6541093a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 --2- 172.21.15.169:0/4052018815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6540ff0d0 0x7fb6540ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 --2- 172.21.15.169:0/4052018815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654102150 0x7fb6541025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 -- 172.21.15.169:0/4052018815 >> 172.21.15.169:0/4052018815 conn(0x7fb6540faa10 msgr2=0x7fb6540fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 -- 172.21.15.169:0/4052018815 shutdown_connections 2024-03-18T20:24:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 -- 172.21.15.169:0/4052018815 wait complete. 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.138+0000 7fb65925a700 1 Processor -- start 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 -- start start 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 0x7fb654101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 0x7fb654100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb654100de0 con 0x7fb654102150 2024-03-18T20:24:14.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb654100f50 con 0x7fb6540ff0d0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65925a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541a0de0 con 0x7fb654102af0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33210/0 (socket says 172.21.15.169:33210) 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 -- 172.21.15.169:0/1963512821 learned_addr learned my addr 172.21.15.169:0/1963512821 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65259c700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 0x7fb654100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb652d9d700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 0x7fb654101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 -- 172.21.15.169:0/1963512821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 msgr2=0x7fb654101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 0x7fb654101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 -- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 msgr2=0x7fb654100100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 0x7fb654100100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 -- 172.21.15.169:0/1963512821 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c009d00 con 0x7fb654102af0 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65259c700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 0x7fb654100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb652d9d700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 0x7fb654101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.139+0000 7fb65359e700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b780 tx=0x7fb64c00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.140+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c00e040 con 0x7fb654102af0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.140+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb64c003e10 con 0x7fb654102af0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.140+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6541a1010 con 0x7fb654102af0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.140+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c005420 con 0x7fb654102af0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.140+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6541a1420 con 0x7fb654102af0 2024-03-18T20:24:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.141+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb654108380 con 0x7fb654102af0 2024-03-18T20:24:14.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.142+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb64c01db10 con 0x7fb654102af0 2024-03-18T20:24:14.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.142+0000 7fb63bfff700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 0x7fb63c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:14.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.142+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb64c0cdd90 con 0x7fb654102af0 2024-03-18T20:24:14.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.142+0000 7fb652d9d700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 0x7fb63c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:14.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.143+0000 7fb652d9d700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 0x7fb63c0a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb654068490 tx=0x7fb644009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.146+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb64c097a30 con 0x7fb654102af0 2024-03-18T20:24:14.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.345+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6540612f0 con 0x7fb63c0a1120 2024-03-18T20:24:14.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.354+0000 7fb63bfff700 1 -- 172.21.15.169:0/1963512821 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb6540612f0 con 0x7fb63c0a1120 2024-03-18T20:24:14.358 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mds.all update... 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 msgr2=0x7fb63c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 0x7fb63c0a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb654068490 tx=0x7fb644009260 comp rx=0 tx=0).stop 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 msgr2=0x7fb654100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b780 tx=0x7fb64c00bd70 comp rx=0 tx=0).stop 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 shutdown_connections 2024-03-18T20:24:14.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb654102af0 0x7fb654100640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb63c0a1120 0x7fb63c0a35e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6540ff0d0 0x7fb654101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 --2- 172.21.15.169:0/1963512821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb654102150 0x7fb654100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 >> 172.21.15.169:0/1963512821 conn(0x7fb6540faa10 msgr2=0x7fb6541054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 shutdown_connections 2024-03-18T20:24:14.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-18T20:24:14.356+0000 7fb65925a700 1 -- 172.21.15.169:0/1963512821 wait complete. 2024-03-18T20:24:14.743 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T20:24:14.745 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T20:24:14.776 INFO:journalctl@ceph.mds.a.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.778 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T20:24:14.804 INFO:journalctl@ceph.mds.d.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.806 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T20:24:14.831 INFO:journalctl@ceph.mds.g.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.832 DEBUG:teuthology.orchestra.run.smithi136:mds.b> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T20:24:14.834 DEBUG:teuthology.orchestra.run.smithi136:mds.e> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T20:24:14.857 INFO:journalctl@ceph.mds.j.smithi098.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.872 INFO:journalctl@ceph.mds.b.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.874 DEBUG:teuthology.orchestra.run.smithi136:mds.h> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T20:24:14.903 INFO:journalctl@ceph.mds.e.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.904 DEBUG:teuthology.orchestra.run.smithi136:mds.k> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T20:24:14.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:14 smithi136 ceph-mon[48073]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:14.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:14.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:14 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:14.928 DEBUG:teuthology.orchestra.run.smithi169:mds.c> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T20:24:14.930 DEBUG:teuthology.orchestra.run.smithi169:mds.f> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T20:24:14.932 INFO:journalctl@ceph.mds.h.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:14 smithi169 ceph-mon[48529]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:14.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:14.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:14 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:14.955 DEBUG:teuthology.orchestra.run.smithi169:mds.i> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T20:24:14.956 INFO:journalctl@ceph.mds.k.smithi136.stdout:-- Logs begin at Mon 2024-03-18 20:03:24 UTC. -- 2024-03-18T20:24:14.962 INFO:journalctl@ceph.mds.c.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:24:14.963 DEBUG:teuthology.orchestra.run.smithi169:mds.l> sudo journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T20:24:14.983 INFO:journalctl@ceph.mds.f.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:24:14.984 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T20:24:14.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:14.990 INFO:journalctl@ceph.mds.i.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:24:15.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:14 smithi098 ceph-mon[43462]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:15.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:15.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:14 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:15.011 INFO:journalctl@ceph.mds.l.smithi169.stdout:-- Logs begin at Mon 2024-03-18 20:03:34 UTC. -- 2024-03-18T20:24:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.134+0000 7f29900a8700 1 -- 172.21.15.98:0/3179245904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 msgr2=0x7f29881011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.134+0000 7f29900a8700 1 --2- 172.21.15.98:0/3179245904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f29881011c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f29840099c0 tx=0x7f2984009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- 172.21.15.98:0/3179245904 shutdown_connections 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- 172.21.15.98:0/3179245904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2988101700 0x7f2988107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- 172.21.15.98:0/3179245904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29880ffb70 0x7f29880fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- 172.21.15.98:0/3179245904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f29881011c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- 172.21.15.98:0/3179245904 >> 172.21.15.98:0/3179245904 conn(0x7f29880fb100 msgr2=0x7f29880fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- 172.21.15.98:0/3179245904 shutdown_connections 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- 172.21.15.98:0/3179245904 wait complete. 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 Processor -- start 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- start start 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29880ffb70 0x7f2988195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2988101700 0x7f298819d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298819d700 con 0x7f2988100d60 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298819d840 con 0x7f2988101700 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f29900a8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298819d9d0 con 0x7f29880ffb70 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f298d643700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f298d643700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41600/0 (socket says 172.21.15.98:41600) 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.135+0000 7f298d643700 1 -- 172.21.15.98:0/2352627924 learned_addr learned my addr 172.21.15.98:0/2352627924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 -- 172.21.15.98:0/2352627924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29880ffb70 msgr2=0x7f2988195bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29880ffb70 0x7f2988195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 -- 172.21.15.98:0/2352627924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2988101700 msgr2=0x7f298819d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2988101700 0x7f298819d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 -- 172.21.15.98:0/2352627924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29840095f0 con 0x7f2988100d60 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f298d643700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2984006010 tx=0x7f29840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298401d070 con 0x7f2988100d60 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f29840040e0 con 0x7f2988100d60 2024-03-18T20:24:15.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984022e60 con 0x7f2988100d60 2024-03-18T20:24:15.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2988069110 con 0x7f2988100d60 2024-03-18T20:24:15.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.136+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29880695f0 con 0x7f2988100d60 2024-03-18T20:24:15.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.137+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2988067180 con 0x7f2988100d60 2024-03-18T20:24:15.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.138+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f2984022600 con 0x7f2988100d60 2024-03-18T20:24:15.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.139+0000 7f297affd700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 0x7f29740a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.139+0000 7f298de44700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 0x7f29740a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.139+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f29840cdd30 con 0x7f2988100d60 2024-03-18T20:24:15.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.139+0000 7f298de44700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 0x7f29740a35e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2988100bc0 tx=0x7f297c006cd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.142+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2984097950 con 0x7f2988100d60 2024-03-18T20:24:15.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.344+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2988106ac0 con 0x7f2988100d60 2024-03-18T20:24:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.345+0000 7f297affd700 1 -- 172.21.15.98:0/2352627924 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18422 (secure 0 0 0) 0x7f298409af70 con 0x7f2988100d60 2024-03-18T20:24:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:15.350 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":77,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:09.636901+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 msgr2=0x7f29740a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 0x7f29740a35e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2988100bc0 tx=0x7f297c006cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 msgr2=0x7f2988196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2984006010 tx=0x7f29840057f0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 shutdown_connections 2024-03-18T20:24:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f29740a1120 0x7f29740a35e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2988101700 0x7f298819d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29880ffb70 0x7f2988195bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 --2- 172.21.15.98:0/2352627924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2988100d60 0x7f2988196110 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 >> 172.21.15.98:0/2352627924 conn(0x7f29880fb100 msgr2=0x7f2988104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 shutdown_connections 2024-03-18T20:24:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.347+0000 7f29900a8700 1 -- 172.21.15.98:0/2352627924 wait complete. 2024-03-18T20:24:15.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:24:15.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/1203954078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041041c0 msgr2=0x7fdb041065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/1203954078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041041c0 0x7fdb041065b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009990 tx=0x7fdaf8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/1203954078 shutdown_connections 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/1203954078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb04106af0 0x7fdb04108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/1203954078 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb04101890 0x7fdb04103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/1203954078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041041c0 0x7fdb041065b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.525+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/1203954078 >> 172.21.15.98:0/1203954078 conn(0x7fdb040fb200 msgr2=0x7fdb040fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/1203954078 shutdown_connections 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/1203954078 wait complete. 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 Processor -- start 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- start start 2024-03-18T20:24:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb041041c0 0x7fdb04197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb04106af0 0x7fdb0419e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0419e760 con 0x7fdb04101890 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0419e8a0 con 0x7fdb041041c0 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.526+0000 7fdb0a8b9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0419ea30 con 0x7fdb04106af0 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41616/0 (socket says 172.21.15.98:41616) 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 -- 172.21.15.98:0/3133524267 learned_addr learned my addr 172.21.15.98:0/3133524267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb037fe700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb041041c0 0x7fdb04197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 -- 172.21.15.98:0/3133524267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb04106af0 msgr2=0x7fdb0419e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb04106af0 0x7fdb0419e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 -- 172.21.15.98:0/3133524267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb041041c0 msgr2=0x7fdb04197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb041041c0 0x7fdb04197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 -- 172.21.15.98:0/3133524267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf80095f0 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb03fff700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400e9a0 tx=0x7fdaf400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4009790 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0419ed10 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdaf400cbd0 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4018680 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.527+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0419f1d0 con 0x7fdb04101890 2024-03-18T20:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.528+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb040fcd80 con 0x7fdb04101890 2024-03-18T20:24:15.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.529+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fdaf400cd40 con 0x7fdb04101890 2024-03-18T20:24:15.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.530+0000 7fdb017fa700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 0x7fdaec0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.530+0000 7fdb037fe700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 0x7fdaec0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.530+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdaf4014070 con 0x7fdb04101890 2024-03-18T20:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.530+0000 7fdb037fe700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 0x7fdaec0a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098e0 tx=0x7fdaf8005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.533+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdaf40965a0 con 0x7fdb04101890 2024-03-18T20:24:15.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.732+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb0402ccb0 con 0x7fdb04101890 2024-03-18T20:24:15.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.732+0000 7fdb017fa700 1 -- 172.21.15.98:0/3133524267 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdaf4099bc0 con 0x7fdb04101890 2024-03-18T20:24:15.737 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:24:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 msgr2=0x7fdaec0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 0x7fdaec0a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80098e0 tx=0x7fdaf8005c80 comp rx=0 tx=0).stop 2024-03-18T20:24:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 msgr2=0x7fdb04196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400e9a0 tx=0x7fdaf400ed60 comp rx=0 tx=0).stop 2024-03-18T20:24:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 shutdown_connections 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdb04106af0 0x7fdb0419e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101890 0x7fdb04196c30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdaec0a1120 0x7fdaec0a35e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 --2- 172.21.15.98:0/3133524267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb041041c0 0x7fdb04197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 >> 172.21.15.98:0/3133524267 conn(0x7fdb040fb200 msgr2=0x7fdb04103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 shutdown_connections 2024-03-18T20:24:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.734+0000 7fdb0a8b9700 1 -- 172.21.15.98:0/3133524267 wait complete. 2024-03-18T20:24:15.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/1570848600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80ffb60 msgr2=0x7fd6c80fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/1570848600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80ffb60 0x7fd6c80fff80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80099c0 tx=0x7fd6b8009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/1570848600 shutdown_connections 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/1570848600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c81016f0 0x7fd6c8107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/1570848600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80ffb60 0x7fd6c80fff80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/1570848600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c8100d50 0x7fd6c81011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/1570848600 >> 172.21.15.98:0/1570848600 conn(0x7fd6c80fb0d0 msgr2=0x7fd6c80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.868+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/1570848600 shutdown_connections 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/1570848600 wait complete. 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 Processor -- start 2024-03-18T20:24:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 -- start start 2024-03-18T20:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c80ffb60 0x7fd6c8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c81016f0 0x7fd6c819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c819e600 con 0x7fd6c8100d50 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c819e770 con 0x7fd6c81016f0 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.869+0000 7fd6ce6e5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c819e930 con 0x7fd6c80ffb60 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41634/0 (socket says 172.21.15.98:41634) 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c80ffb60 0x7fd6c8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 -- 172.21.15.98:0/404880218 learned_addr learned my addr 172.21.15.98:0/404880218 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6ccc82700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c81016f0 0x7fd6c819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 -- 172.21.15.98:0/404880218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c80ffb60 msgr2=0x7fd6c8068930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c80ffb60 0x7fd6c8068930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 -- 172.21.15.98:0/404880218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c81016f0 msgr2=0x7fd6c819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c81016f0 0x7fd6c819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 -- 172.21.15.98:0/404880218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80095f0 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c77fe700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c9b0 tx=0x7fd6b000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0007b00 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd6b0007c60 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c819ebb0 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0018660 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.870+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c819f110 con 0x7fd6c8100d50 2024-03-18T20:24:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.871+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c8106f80 con 0x7fd6c8100d50 2024-03-18T20:24:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.872+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd6b000f450 con 0x7fd6c8100d50 2024-03-18T20:24:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.873+0000 7fd6c57fa700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 0x7fd6b40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.873+0000 7fd6c7fff700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 0x7fd6b40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.873+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd6b00cc6d0 con 0x7fd6c8100d50 2024-03-18T20:24:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.873+0000 7fd6c7fff700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 0x7fd6b40a36e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009990 tx=0x7fd6b8003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:15.876+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd6b0096370 con 0x7fd6c8100d50 2024-03-18T20:24:15.978 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: Deploying daemon mds.l on smithi169 2024-03-18T20:24:15.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:15 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2352627924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:24:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-03-18T20:24:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:15.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: Deploying daemon mds.l on smithi169 2024-03-18T20:24:15.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:15 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2352627924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: Deploying daemon mds.l on smithi169 2024-03-18T20:24:16.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:15 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2352627924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:16.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.117+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6c8067a10 con 0x7fd6c8100d50 2024-03-18T20:24:16.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.117+0000 7fd6c57fa700 1 -- 172.21.15.98:0/404880218 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fd6b000fe60 con 0x7fd6c8100d50 2024-03-18T20:24:16.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:16.122 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 msgr2=0x7fd6b40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 0x7fd6b40a36e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009990 tx=0x7fd6b8003680 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 msgr2=0x7fd6c819b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c9b0 tx=0x7fd6b000ccc0 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 shutdown_connections 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6b40a1220 0x7fd6b40a36e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c81016f0 0x7fd6c819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6c80ffb60 0x7fd6c8068930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 --2- 172.21.15.98:0/404880218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c8100d50 0x7fd6c819b7f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 >> 172.21.15.98:0/404880218 conn(0x7fd6c80fb0d0 msgr2=0x7fd6c81040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 shutdown_connections 2024-03-18T20:24:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:16.119+0000 7fd6ce6e5700 1 -- 172.21.15.98:0/404880218 wait complete. 2024-03-18T20:24:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-03-18T20:24:16.317 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:24:15 smithi169 systemd[1]: Starting Ceph mds.l for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:16.634 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mds[85190]: -- 172.21.15.169:0/790534339 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558da75441a0 con 0x558da751ac00 2024-03-18T20:24:16.634 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-l[85167]: starting mds.l at 2024-03-18T20:24:16.634 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:24:16 smithi169 bash[85153]: 8bed39dce3da55e4fa52263c9ac581057a7fc208c2cf8ee3158659f43db8d755 2024-03-18T20:24:16.634 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:24:16 smithi169 systemd[1]: Started Ceph mds.l for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3133524267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/404880218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.169:6832/1320002530,v1:172.21.15.169:6833/1320002530] up:boot 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: fsmap 1 up:standby 2024-03-18T20:24:16.956 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:16.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:16 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3133524267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/404880218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.169:6832/1320002530,v1:172.21.15.169:6833/1320002530] up:boot 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: fsmap 1 up:standby 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:17.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:17.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:16 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3133524267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/404880218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.169:6832/1320002530,v1:172.21.15.169:6833/1320002530] up:boot 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: fsmap 1 up:standby 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:17.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:17.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:17.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:16 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:17.923 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:24:17 smithi136 systemd[1]: Starting Ceph mds.k for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:17.923 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:17 smithi136 ceph-mon[48073]: Deploying daemon mds.k on smithi136 2024-03-18T20:24:18.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:17 smithi098 ceph-mon[43462]: Deploying daemon mds.k on smithi136 2024-03-18T20:24:18.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:17 smithi169 ceph-mon[48529]: Deploying daemon mds.k on smithi136 2024-03-18T20:24:18.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 -- 172.21.15.98:0/1151604411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c101e70 msgr2=0x7fd71c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 --2- 172.21.15.98:0/1151604411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c101e70 0x7fd71c1022d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd7100099c0 tx=0x7fd710009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 -- 172.21.15.98:0/1151604411 shutdown_connections 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 --2- 172.21.15.98:0/1151604411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c102810 0x7fd71c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 --2- 172.21.15.98:0/1151604411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c100c80 0x7fd71c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 --2- 172.21.15.98:0/1151604411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c101e70 0x7fd71c1022d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 -- 172.21.15.98:0/1151604411 >> 172.21.15.98:0/1151604411 conn(0x7fd71c0fc1f0 msgr2=0x7fd71c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 -- 172.21.15.98:0/1151604411 shutdown_connections 2024-03-18T20:24:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.267+0000 7fd720cb5700 1 -- 172.21.15.98:0/1151604411 wait complete. 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 Processor -- start 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 -- start start 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101e70 0x7fd71c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c102810 0x7fd71c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c073e80 con 0x7fd71c100c80 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c073ff0 con 0x7fd71c101e70 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd720cb5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c1a0b70 con 0x7fd71c102810 2024-03-18T20:24:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd71a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd71a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41650/0 (socket says 172.21.15.98:41650) 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd71a59c700 1 -- 172.21.15.98:0/3229097770 learned_addr learned my addr 172.21.15.98:0/3229097770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.268+0000 7fd719d9b700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101e70 0x7fd71c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71ad9d700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c102810 0x7fd71c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 -- 172.21.15.98:0/3229097770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c102810 msgr2=0x7fd71c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c102810 0x7fd71c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 -- 172.21.15.98:0/3229097770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101e70 msgr2=0x7fd71c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101e70 0x7fd71c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 -- 172.21.15.98:0/3229097770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100095f0 con 0x7fd71c100c80 2024-03-18T20:24:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd71a59c700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e970 tx=0x7fd70c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c009790 con 0x7fd71c100c80 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd70c00eec0 con 0x7fd71c100c80 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71c1a0e00 con 0x7fd71c100c80 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c003c10 con 0x7fd71c100c80 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.269+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c1a1220 con 0x7fd71c100c80 2024-03-18T20:24:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.270+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71c108120 con 0x7fd71c100c80 2024-03-18T20:24:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.271+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd70c00cb70 con 0x7fd71c100c80 2024-03-18T20:24:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.272+0000 7fd70b7fe700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 0x7fd7040a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.272+0000 7fd719d9b700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 0x7fd7040a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.272+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd70c014070 con 0x7fd71c100c80 2024-03-18T20:24:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.272+0000 7fd719d9b700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 0x7fd7040a35e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd710000c00 tx=0x7fd710003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:18.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.276+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd70c0962c0 con 0x7fd71c100c80 2024-03-18T20:24:18.491 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:24:18 smithi136 bash[83626]: 6003bbe0872da388c6e39b9240d0eb35c2ef502f346e63d150547cf8b57368c9 2024-03-18T20:24:18.491 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:24:18 smithi136 systemd[1]: Started Ceph mds.k for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:18.491 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mds[83664]: -- 172.21.15.136:0/4290316214 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d8a6201a0 con 0x557d8a5f7000 2024-03-18T20:24:18.491 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-k[83641]: starting mds.k at 2024-03-18T20:24:18.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.545+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd71c02cf30 con 0x7fd71c100c80 2024-03-18T20:24:18.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.546+0000 7fd70b7fe700 1 -- 172.21.15.98:0/3229097770 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fd70c0998e0 con 0x7fd71c100c80 2024-03-18T20:24:18.551 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:18.551 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 msgr2=0x7fd7040a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 0x7fd7040a35e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd710000c00 tx=0x7fd710003680 comp rx=0 tx=0).stop 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 msgr2=0x7fd71c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e970 tx=0x7fd70c00ec80 comp rx=0 tx=0).stop 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 shutdown_connections 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71c102810 0x7fd71c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd71c100c80 0x7fd71c074b10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7040a1120 0x7fd7040a35e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 --2- 172.21.15.98:0/3229097770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101e70 0x7fd71c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 >> 172.21.15.98:0/3229097770 conn(0x7fd71c0fc1f0 msgr2=0x7fd71c105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 shutdown_connections 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:18.548+0000 7fd720cb5700 1 -- 172.21.15.98:0/3229097770 wait complete. 2024-03-18T20:24:18.554 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:18.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:18 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3229097770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:18.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:18.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:18 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3229097770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:19.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:18 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3229097770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:19.593 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:24:19 smithi098 systemd[1]: Starting Ceph mds.j for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:19 smithi136 ceph-mon[48073]: Deploying daemon mds.j on smithi098 2024-03-18T20:24:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:19 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6832/1211632092,v1:172.21.15.136:6833/1211632092] up:boot 2024-03-18T20:24:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:19 smithi136 ceph-mon[48073]: fsmap 2 up:standby 2024-03-18T20:24:19.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:19 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T20:24:20.092 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-mds[93547]: -- 172.21.15.98:0/298755532 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557336ac41a0 con 0x557336a9b000 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-j[93524]: starting mds.j at 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:24:19 smithi098 bash[93509]: 617961d0f97cb335e9376306d49c08d068da47f2a22ddbd85bc0401ed2bc0cb9 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:24:19 smithi098 systemd[1]: Started Ceph mds.j for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-mon[43462]: Deploying daemon mds.j on smithi098 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6832/1211632092,v1:172.21.15.136:6833/1211632092] up:boot 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-mon[43462]: fsmap 2 up:standby 2024-03-18T20:24:20.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:19 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T20:24:20.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:19 smithi169 ceph-mon[48529]: Deploying daemon mds.j on smithi098 2024-03-18T20:24:20.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:19 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6832/1211632092,v1:172.21.15.136:6833/1211632092] up:boot 2024-03-18T20:24:20.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:19 smithi169 ceph-mon[48529]: fsmap 2 up:standby 2024-03-18T20:24:20.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:19 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T20:24:20.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 -- 172.21.15.98:0/850047085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 msgr2=0x7fcc68109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/850047085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc68109120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000b0a0 tx=0x7fcc6000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 -- 172.21.15.98:0/850047085 shutdown_connections 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/850047085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc68109120 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/850047085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc680fef00 0x7fcc680ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/850047085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc68101ed0 0x7fcc68102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 -- 172.21.15.98:0/850047085 >> 172.21.15.98:0/850047085 conn(0x7fcc680fa850 msgr2=0x7fcc680fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 -- 172.21.15.98:0/850047085 shutdown_connections 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.695+0000 7fcc6ece8700 1 -- 172.21.15.98:0/850047085 wait complete. 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 Processor -- start 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 -- start start 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 0x7fcc68196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68101ed0 0x7fcc681970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6819e6c0 con 0x7fcc68102870 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6819e800 con 0x7fcc68101ed0 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ece8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6819e990 con 0x7fcc680fef00 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41670/0 (socket says 172.21.15.98:41670) 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 -- 172.21.15.98:0/612629368 learned_addr learned my addr 172.21.15.98:0/612629368 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 -- 172.21.15.98:0/612629368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 msgr2=0x7fcc68196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ca84700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 0x7fcc68196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 0x7fcc68196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 -- 172.21.15.98:0/612629368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68101ed0 msgr2=0x7fcc681970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68101ed0 0x7fcc681970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6d285700 1 -- 172.21.15.98:0/612629368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc60009d00 con 0x7fcc68102870 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.696+0000 7fcc6ca84700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 0x7fcc68196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc6d285700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000b830 tx=0x7fcc60004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6000e040 con 0x7fcc68102870 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcc6001dd20 con 0x7fcc68102870 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6819ec10 con 0x7fcc68102870 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6001bcc0 con 0x7fcc68102870 2024-03-18T20:24:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.697+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6819f110 con 0x7fcc68102870 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.698+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc68108090 con 0x7fcc68102870 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.699+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fcc6001b450 con 0x7fcc68102870 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.699+0000 7fcc65ffb700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 0x7fcc580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.700+0000 7fcc6ca84700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 0x7fcc580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.700+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcc600cec60 con 0x7fcc68102870 2024-03-18T20:24:20.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.700+0000 7fcc6ca84700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 0x7fcc580a35e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc68068fe0 tx=0x7fcc5c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:20.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.703+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcc60098920 con 0x7fcc68102870 2024-03-18T20:24:20.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: Deploying daemon mds.d on smithi098 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.98:6834/2703259381,v1:172.21.15.98:6835/2703259381] up:boot 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: fsmap 3 up:standby 2024-03-18T20:24:20.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:20 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T20:24:20.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.961+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc68067df0 con 0x7fcc68102870 2024-03-18T20:24:20.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.962+0000 7fcc65ffb700 1 -- 172.21.15.98:0/612629368 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fcc6009bf40 con 0x7fcc68102870 2024-03-18T20:24:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:20.966 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 msgr2=0x7fcc580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 0x7fcc580a35e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc68068fe0 tx=0x7fcc5c008040 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 msgr2=0x7fcc6819e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000b830 tx=0x7fcc60004d30 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 shutdown_connections 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc68102870 0x7fcc6819e0f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc680fef00 0x7fcc68196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcc580a1120 0x7fcc580a35e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 --2- 172.21.15.98:0/612629368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68101ed0 0x7fcc681970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 >> 172.21.15.98:0/612629368 conn(0x7fcc680fa850 msgr2=0x7fcc681051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 shutdown_connections 2024-03-18T20:24:20.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:20.966+0000 7fcc6ece8700 1 -- 172.21.15.98:0/612629368 wait complete. 2024-03-18T20:24:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 4 2024-03-18T20:24:21.071 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:24:21 smithi098 systemd[1]: Starting Ceph mds.d for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: Deploying daemon mds.d on smithi098 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.98:6834/2703259381,v1:172.21.15.98:6835/2703259381] up:boot 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: fsmap 3 up:standby 2024-03-18T20:24:21.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:20 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: Deploying daemon mds.d on smithi098 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.98:6834/2703259381,v1:172.21.15.98:6835/2703259381] up:boot 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: fsmap 3 up:standby 2024-03-18T20:24:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:20 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T20:24:21.819 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mds[93933]: -- 172.21.15.98:0/1955116736 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d1cdce1a0 con 0x561d1cda4800 2024-03-18T20:24:21.819 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-d[93909]: starting mds.d at 2024-03-18T20:24:21.820 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:24:21 smithi098 bash[93895]: 5880f643492ef331493f4085b02e92afd023346972b44cb0eefb5cd23d72560f 2024-03-18T20:24:21.820 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:24:21 smithi098 systemd[1]: Started Ceph mds.d for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:22.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/612629368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:22.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:21 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/612629368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:22.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:21 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:22.212 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/612629368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:22.213 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:21 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:22.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:23.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:22 smithi098 ceph-mon[43462]: Deploying daemon mds.e on smithi136 2024-03-18T20:24:23.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:22 smithi098 ceph-mon[43462]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:23.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:22 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.98:6836/1932954082,v1:172.21.15.98:6837/1932954082] up:boot 2024-03-18T20:24:23.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:22 smithi098 ceph-mon[43462]: fsmap 4 up:standby 2024-03-18T20:24:23.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:22 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T20:24:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 -- 172.21.15.98:0/2516710300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e40ffb60 msgr2=0x7f23e40fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 --2- 172.21.15.98:0/2516710300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e40ffb60 0x7f23e40fff80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23e00099c0 tx=0x7f23e0009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 -- 172.21.15.98:0/2516710300 shutdown_connections 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 --2- 172.21.15.98:0/2516710300 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e41016f0 0x7f23e4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 --2- 172.21.15.98:0/2516710300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e40ffb60 0x7f23e40fff80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 --2- 172.21.15.98:0/2516710300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e4100d50 0x7f23e41011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 -- 172.21.15.98:0/2516710300 >> 172.21.15.98:0/2516710300 conn(0x7f23e40fb0d0 msgr2=0x7f23e40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 -- 172.21.15.98:0/2516710300 shutdown_connections 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.126+0000 7f23ec315700 1 -- 172.21.15.98:0/2516710300 wait complete. 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 Processor -- start 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 -- start start 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 0x7f23e4195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 0x7f23e4195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e419d5d0 con 0x7f23e41016f0 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e419d710 con 0x7f23e4100d50 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ec315700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e419d8a0 con 0x7f23e40ffb60 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ea8b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ea8b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41706/0 (socket says 172.21.15.98:41706) 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.127+0000 7f23ea8b2700 1 -- 172.21.15.98:0/1699518967 learned_addr learned my addr 172.21.15.98:0/1699518967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea0b1700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 0x7f23e4195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23e98b0700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 0x7f23e4195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 -- 172.21.15.98:0/1699518967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 msgr2=0x7f23e4195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 0x7f23e4195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 -- 172.21.15.98:0/1699518967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 msgr2=0x7f23e4195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 0x7f23e4195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 -- 172.21.15.98:0/1699518967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00095f0 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea0b1700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 0x7f23e4195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23e98b0700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 0x7f23e4195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ea8b2700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00ada0 tx=0x7f23dc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc00f070 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f23dc007b30 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e419db80 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0053e0 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.128+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e419e090 con 0x7f23e41016f0 2024-03-18T20:24:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.129+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e4107080 con 0x7f23e41016f0 2024-03-18T20:24:23.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:22 smithi169 ceph-mon[48529]: Deploying daemon mds.e on smithi136 2024-03-18T20:24:23.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:22 smithi169 ceph-mon[48529]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:23.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:22 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.98:6836/1932954082,v1:172.21.15.98:6837/1932954082] up:boot 2024-03-18T20:24:23.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:22 smithi169 ceph-mon[48529]: fsmap 4 up:standby 2024-03-18T20:24:23.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:22 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T20:24:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.130+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f23dc00c980 con 0x7f23e41016f0 2024-03-18T20:24:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.130+0000 7f23d77fe700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 0x7f23d00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.131+0000 7f23ea0b1700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 0x7f23d00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.131+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f23dc0cbb10 con 0x7f23e41016f0 2024-03-18T20:24:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.131+0000 7f23ea0b1700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 0x7f23d00a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f23e0003780 tx=0x7f23e0009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.135+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f23dc0957d0 con 0x7f23e41016f0 2024-03-18T20:24:23.140 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:22 smithi136 ceph-mon[48073]: Deploying daemon mds.e on smithi136 2024-03-18T20:24:23.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:22 smithi136 ceph-mon[48073]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:23.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:22 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.98:6836/1932954082,v1:172.21.15.98:6837/1932954082] up:boot 2024-03-18T20:24:23.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:22 smithi136 ceph-mon[48073]: fsmap 4 up:standby 2024-03-18T20:24:23.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:22 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T20:24:23.141 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:24:22 smithi136 systemd[1]: Starting Ceph mds.e for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:23.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.380+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23e4067a10 con 0x7f23e41016f0 2024-03-18T20:24:23.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.380+0000 7f23d77fe700 1 -- 172.21.15.98:0/1699518967 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7f23dc017de0 con 0x7f23e41016f0 2024-03-18T20:24:23.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:23.384 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 msgr2=0x7f23d00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 0x7f23d00a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f23e0003780 tx=0x7f23e0009f10 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 msgr2=0x7f23e419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00ada0 tx=0x7f23dc00c330 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 shutdown_connections 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e41016f0 0x7f23e419d000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23e40ffb60 0x7f23e4195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f23d00a11a0 0x7f23d00a3660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 --2- 172.21.15.98:0/1699518967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23e4100d50 0x7f23e4195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 >> 172.21.15.98:0/1699518967 conn(0x7f23e40fb0d0 msgr2=0x7f23e41041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 shutdown_connections 2024-03-18T20:24:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:23.382+0000 7f23ec315700 1 -- 172.21.15.98:0/1699518967 wait complete. 2024-03-18T20:24:23.388 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-03-18T20:24:23.741 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:24:23 smithi136 ceph-mds[84006]: -- 172.21.15.136:0/1945811611 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e84d5181a0 con 0x55e84d4eec00 2024-03-18T20:24:23.741 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:24:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-e[83983]: starting mds.e at 2024-03-18T20:24:23.741 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:24:23 smithi136 bash[83968]: 47f3953132487573c54816ec1ad11ad50c9255d497d5a377bcc5d72f52098f29 2024-03-18T20:24:23.741 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:24:23 smithi136 systemd[1]: Started Ceph mds.e for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:24.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: Deploying daemon mds.f on smithi169 2024-03-18T20:24:24.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1699518967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:24.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:24.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6834/4136937912,v1:172.21.15.136:6835/4136937912] up:boot 2024-03-18T20:24:24.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: fsmap 5 up:standby 2024-03-18T20:24:24.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:24 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: Deploying daemon mds.f on smithi169 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1699518967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6834/4136937912,v1:172.21.15.136:6835/4136937912] up:boot 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: fsmap 5 up:standby 2024-03-18T20:24:24.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:24 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: Deploying daemon mds.f on smithi169 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1699518967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:24.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6834/4136937912,v1:172.21.15.136:6835/4136937912] up:boot 2024-03-18T20:24:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: fsmap 5 up:standby 2024-03-18T20:24:24.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:24 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T20:24:24.936 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:24:24 smithi169 systemd[1]: Starting Ceph mds.f for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:25.384 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:24:25 smithi169 ceph-mds[85532]: -- 172.21.15.169:0/3973184148 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5637de8a81a0 con 0x5637de87e800 2024-03-18T20:24:25.384 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:24:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-f[85509]: starting mds.f at 2024-03-18T20:24:25.384 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:24:25 smithi169 bash[85494]: 6387423f4382db075acfb19b7b33205f447c7f6888b30e78407d438070ede1a6 2024-03-18T20:24:25.384 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:24:25 smithi169 systemd[1]: Started Ceph mds.f for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:25.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:25.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 -- 172.21.15.98:0/1269137281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270101e70 msgr2=0x7fc2701022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:25.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 --2- 172.21.15.98:0/1269137281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270101e70 0x7fc2701022d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0099c0 tx=0x7fc25c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 -- 172.21.15.98:0/1269137281 shutdown_connections 2024-03-18T20:24:25.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 --2- 172.21.15.98:0/1269137281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270102810 0x7fc2701090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 --2- 172.21.15.98:0/1269137281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 0x7fc2701010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 --2- 172.21.15.98:0/1269137281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270101e70 0x7fc2701022d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 -- 172.21.15.98:0/1269137281 >> 172.21.15.98:0/1269137281 conn(0x7fc2700fc210 msgr2=0x7fc2700fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 -- 172.21.15.98:0/1269137281 shutdown_connections 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 -- 172.21.15.98:0/1269137281 wait complete. 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.529+0000 7fc276c58700 1 Processor -- start 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 -- start start 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 0x7fc270196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270101e70 0x7fc270197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27019e600 con 0x7fc270102810 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27019e740 con 0x7fc270100c80 2024-03-18T20:24:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27019e8d0 con 0x7fc270101e70 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49704/0 (socket says 172.21.15.98:49704) 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 -- 172.21.15.98:0/3357977666 learned_addr learned my addr 172.21.15.98:0/3357977666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc26ffff700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270101e70 0x7fc270197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2749f4700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 0x7fc270196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 -- 172.21.15.98:0/3357977666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270101e70 msgr2=0x7fc270197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270101e70 0x7fc270197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 -- 172.21.15.98:0/3357977666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 msgr2=0x7fc270196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 0x7fc270196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 -- 172.21.15.98:0/3357977666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0095f0 con 0x7fc270102810 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc2751f5700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc26800ef30 tx=0x7fc26800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268018070 con 0x7fc270102810 2024-03-18T20:24:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc26800f040 con 0x7fc270102810 2024-03-18T20:24:25.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27019eb50 con 0x7fc270102810 2024-03-18T20:24:25.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.530+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cbc0 con 0x7fc270102810 2024-03-18T20:24:25.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.531+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27019f010 con 0x7fc270102810 2024-03-18T20:24:25.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.531+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2701080d0 con 0x7fc270102810 2024-03-18T20:24:25.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.533+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc26800cd60 con 0x7fc270102810 2024-03-18T20:24:25.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.533+0000 7fc26dffb700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 0x7fc2600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:25.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.533+0000 7fc2749f4700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 0x7fc2600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:25.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.533+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc2680cc4c0 con 0x7fc270102810 2024-03-18T20:24:25.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.534+0000 7fc2749f4700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 0x7fc2600a35e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc270101cd0 tx=0x7fc26400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:25.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.537+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc268096180 con 0x7fc270102810 2024-03-18T20:24:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.778+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc270067180 con 0x7fc270102810 2024-03-18T20:24:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.779+0000 7fc26dffb700 1 -- 172.21.15.98:0/3357977666 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7fc2680997a0 con 0x7fc270102810 2024-03-18T20:24:25.783 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:25.783 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2024-03-18T20:24:25.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 msgr2=0x7fc2600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 0x7fc2600a35e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc270101cd0 tx=0x7fc26400a3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 msgr2=0x7fc27019e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc26800ef30 tx=0x7fc26800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 shutdown_connections 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc270102810 0x7fc27019e030 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2600a1120 0x7fc2600a35e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc270100c80 0x7fc270196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 --2- 172.21.15.98:0/3357977666 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc270101e70 0x7fc270197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 >> 172.21.15.98:0/3357977666 conn(0x7fc2700fc210 msgr2=0x7fc270105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 shutdown_connections 2024-03-18T20:24:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:25.781+0000 7fc276c58700 1 -- 172.21.15.98:0/3357977666 wait complete. 2024-03-18T20:24:25.786 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-03-18T20:24:26.259 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: Deploying daemon mds.c on smithi169 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.169:6834/477760816,v1:172.21.15.169:6835/477760816] up:boot 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: fsmap 6 up:standby 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:26.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3357977666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:26.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: Deploying daemon mds.c on smithi169 2024-03-18T20:24:26.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.169:6834/477760816,v1:172.21.15.169:6835/477760816] up:boot 2024-03-18T20:24:26.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: fsmap 6 up:standby 2024-03-18T20:24:26.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T20:24:26.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:26.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3357977666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: Deploying daemon mds.c on smithi169 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.169:6834/477760816,v1:172.21.15.169:6835/477760816] up:boot 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: fsmap 6 up:standby 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T20:24:26.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:26.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3357977666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:27.383 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:24:26 smithi169 bash[85836]: dcdda2bc0b1fac0b52fdb9f6649df70b016a5768e3dbc947379d2def23088814 2024-03-18T20:24:27.384 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:24:26 smithi169 systemd[1]: Started Ceph mds.c for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:27.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 -- 172.21.15.98:0/1322984270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d80ffb60 msgr2=0x7ff2d80fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 --2- 172.21.15.98:0/1322984270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80fff80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff2d40099c0 tx=0x7ff2d4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 -- 172.21.15.98:0/1322984270 shutdown_connections 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 --2- 172.21.15.98:0/1322984270 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d81016f0 0x7ff2d8107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 --2- 172.21.15.98:0/1322984270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80fff80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 --2- 172.21.15.98:0/1322984270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d8100d50 0x7ff2d81011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 -- 172.21.15.98:0/1322984270 >> 172.21.15.98:0/1322984270 conn(0x7ff2d80fb0d0 msgr2=0x7ff2d80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 -- 172.21.15.98:0/1322984270 shutdown_connections 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.928+0000 7ff2df344700 1 -- 172.21.15.98:0/1322984270 wait complete. 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 Processor -- start 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 -- start start 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80688c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 0x7ff2d819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d819e600 con 0x7ff2d8100d50 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d819e770 con 0x7ff2d81016f0 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2df344700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d819e930 con 0x7ff2d80ffb60 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:27.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49714/0 (socket says 172.21.15.98:49714) 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 -- 172.21.15.98:0/3799846995 learned_addr learned my addr 172.21.15.98:0/3799846995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dd0e0700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dd8e1700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 0x7ff2d819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 -- 172.21.15.98:0/3799846995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d80ffb60 msgr2=0x7ff2d80688c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80688c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 -- 172.21.15.98:0/3799846995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 msgr2=0x7ff2d819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 0x7ff2d819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 -- 172.21.15.98:0/3799846995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d40095f0 con 0x7ff2d8100d50 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dd8e1700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 0x7ff2d819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.929+0000 7ff2dc8df700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00e9a0 tx=0x7ff2cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.930+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc009790 con 0x7ff2d8100d50 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.930+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff2cc00cbd0 con 0x7ff2d8100d50 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.930+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d819ec10 con 0x7ff2d8100d50 2024-03-18T20:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.930+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc018680 con 0x7ff2d8100d50 2024-03-18T20:24:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.930+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d819f170 con 0x7ff2d8100d50 2024-03-18T20:24:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.931+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d8107070 con 0x7ff2d8100d50 2024-03-18T20:24:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.932+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ff2cc00cd40 con 0x7ff2d8100d50 2024-03-18T20:24:27.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.932+0000 7ff2ca7fc700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 0x7ff2c40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.932+0000 7ff2dd0e0700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 0x7ff2c40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.932+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2cc014070 con 0x7ff2d8100d50 2024-03-18T20:24:27.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.933+0000 7ff2dd0e0700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 0x7ff2c40a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4006010 tx=0x7ff2d400bfd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:27.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:27.937+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff2cc0529f0 con 0x7ff2d8100d50 2024-03-18T20:24:28.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: Deploying daemon mds.h on smithi136 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.169:6836/1340440313,v1:172.21.15.169:6837/1340440313] up:boot 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: fsmap 7 up:standby 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T20:24:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:27 smithi098 ceph-mon[43462]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.132 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: Deploying daemon mds.h on smithi136 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.169:6836/1340440313,v1:172.21.15.169:6837/1340440313] up:boot 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: fsmap 7 up:standby 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T20:24:28.133 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:27 smithi136 ceph-mon[48073]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.179+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2d8067a10 con 0x7ff2d8100d50 2024-03-18T20:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.180+0000 7ff2ca7fc700 1 -- 172.21.15.98:0/3799846995 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7ff2cc096220 con 0x7ff2d8100d50 2024-03-18T20:24:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:28.184 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 msgr2=0x7ff2c40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 0x7ff2c40a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4006010 tx=0x7ff2d400bfd0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 msgr2=0x7ff2d819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00e9a0 tx=0x7ff2cc00ecb0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 shutdown_connections 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff2c40a11a0 0x7ff2c40a3660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2d81016f0 0x7ff2d819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d80ffb60 0x7ff2d80688c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 --2- 172.21.15.98:0/3799846995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2d8100d50 0x7ff2d819b8d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 >> 172.21.15.98:0/3799846995 conn(0x7ff2d80fb0d0 msgr2=0x7ff2d81041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 shutdown_connections 2024-03-18T20:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:28.182+0000 7ff2df344700 1 -- 172.21.15.98:0/3799846995 wait complete. 2024-03-18T20:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-03-18T20:24:28.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: Deploying daemon mds.h on smithi136 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.169:6836/1340440313,v1:172.21.15.169:6837/1340440313] up:boot 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: fsmap 7 up:standby 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T20:24:28.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:27 smithi169 ceph-mon[48529]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:28.387 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:24:28 smithi136 systemd[1]: Starting Ceph mds.h for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3799846995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:29.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:28 smithi169 ceph-mon[48529]: Deploying daemon mds.i on smithi169 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3799846995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mon[48073]: Deploying daemon mds.i on smithi169 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-mds[84350]: -- 172.21.15.136:0/100746284 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56002423a1a0 con 0x560024210c00 2024-03-18T20:24:29.241 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:24:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-h[84327]: starting mds.h at 2024-03-18T20:24:29.242 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:24:28 smithi136 bash[84312]: b8b747e34d314bc772d24225b892c613fcf142a713c2dee381fca8076b6ea6fe 2024-03-18T20:24:29.242 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:24:28 smithi136 systemd[1]: Started Ceph mds.h for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3799846995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:29.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:28 smithi098 ceph-mon[43462]: Deploying daemon mds.i on smithi169 2024-03-18T20:24:30.078 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:24:30 smithi169 systemd[1]: Starting Ceph mds.i for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:30.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 -- 172.21.15.98:0/1770771982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8106940 msgr2=0x7f60d8108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 --2- 172.21.15.98:0/1770771982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8106940 0x7f60d8108db0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f60d000b0a0 tx=0x7f60d000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 -- 172.21.15.98:0/1770771982 shutdown_connections 2024-03-18T20:24:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 --2- 172.21.15.98:0/1770771982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8106940 0x7f60d8108db0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 --2- 172.21.15.98:0/1770771982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d81016e0 0x7f60d8103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 --2- 172.21.15.98:0/1770771982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d8104010 0x7f60d8106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 -- 172.21.15.98:0/1770771982 >> 172.21.15.98:0/1770771982 conn(0x7f60d80fb0d0 msgr2=0x7f60d80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 -- 172.21.15.98:0/1770771982 shutdown_connections 2024-03-18T20:24:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.330+0000 7f60dd32b700 1 -- 172.21.15.98:0/1770771982 wait complete. 2024-03-18T20:24:30.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 Processor -- start 2024-03-18T20:24:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 -- start start 2024-03-18T20:24:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d81016e0 0x7f60d8196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d8106940 0x7f60d819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:30.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d819e6c0 con 0x7f60d8104010 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d819e800 con 0x7f60d81016e0 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60dd32b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d819e990 con 0x7f60d8106940 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60d67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60d67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49736/0 (socket says 172.21.15.98:49736) 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60d67fc700 1 -- 172.21.15.98:0/3456029334 learned_addr learned my addr 172.21.15.98:0/3456029334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.331+0000 7f60d77fe700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d8106940 0x7f60d819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d81016e0 0x7f60d8196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 -- 172.21.15.98:0/3456029334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d8106940 msgr2=0x7f60d819e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d8106940 0x7f60d819e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 -- 172.21.15.98:0/3456029334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d81016e0 msgr2=0x7f60d8196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:30.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d81016e0 0x7f60d8196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 -- 172.21.15.98:0/3456029334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d0009d00 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60d67fc700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60c800d980 tx=0x7f60c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8004220 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d819ec70 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f60c8007640 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c80055e0 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.332+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d819f1d0 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.333+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d80fcc50 con 0x7f60d8104010 2024-03-18T20:24:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.334+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f60c8004750 con 0x7f60d8104010 2024-03-18T20:24:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.334+0000 7f60bffff700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 0x7f60c00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.335+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60c80cc6e0 con 0x7f60d8104010 2024-03-18T20:24:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.335+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 0x7f60c00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.335+0000 7f60d6ffd700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 0x7f60c00a35e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f60cc005950 tx=0x7f60cc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:30.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.338+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60c8096310 con 0x7f60d8104010 2024-03-18T20:24:30.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:30 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:boot 2024-03-18T20:24:30.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:30 smithi098 ceph-mon[43462]: fsmap 8 up:standby 2024-03-18T20:24:30.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:30 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T20:24:30.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:30 smithi098 ceph-mon[43462]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.590+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d8067a10 con 0x7f60d8104010 2024-03-18T20:24:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.590+0000 7f60bffff700 1 -- 172.21.15.98:0/3456029334 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f60c8005740 con 0x7f60d8104010 2024-03-18T20:24:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:30.594 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 msgr2=0x7f60c00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 0x7f60c00a35e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f60cc005950 tx=0x7f60cc0058e0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 msgr2=0x7f60d81970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60c800d980 tx=0x7f60c800dc90 comp rx=0 tx=0).stop 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 shutdown_connections 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60d8106940 0x7f60d819e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60c00a1120 0x7f60c00a35e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d81016e0 0x7f60d8196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 --2- 172.21.15.98:0/3456029334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60d8104010 0x7f60d81970d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 >> 172.21.15.98:0/3456029334 conn(0x7f60d80fb0d0 msgr2=0x7f60d8101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 shutdown_connections 2024-03-18T20:24:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:30.593+0000 7f60dd32b700 1 -- 172.21.15.98:0/3456029334 wait complete. 2024-03-18T20:24:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-03-18T20:24:30.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:30 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:boot 2024-03-18T20:24:30.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:30 smithi136 ceph-mon[48073]: fsmap 8 up:standby 2024-03-18T20:24:30.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:30 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T20:24:30.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:30 smithi136 ceph-mon[48073]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:30.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:boot 2024-03-18T20:24:30.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-mon[48529]: fsmap 8 up:standby 2024-03-18T20:24:30.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T20:24:30.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-mon[48529]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:30.884 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-mds[86214]: -- 172.21.15.169:0/1635604902 <== mon.1 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558dc369c1a0 con 0x558dc3673000 2024-03-18T20:24:30.885 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:24:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-i[86191]: starting mds.i at 2024-03-18T20:24:30.885 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:24:30 smithi169 bash[86176]: d42883216776cdefa2e38df7c8e473f0fa4376ff4923c9263867691ced0670c2 2024-03-18T20:24:30.885 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:24:30 smithi169 systemd[1]: Started Ceph mds.i for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:31.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: Deploying daemon mds.g on smithi098 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3456029334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:boot 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: fsmap 9 up:standby 2024-03-18T20:24:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:31 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: Deploying daemon mds.g on smithi098 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3456029334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:boot 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: fsmap 9 up:standby 2024-03-18T20:24:31.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:31 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: Deploying daemon mds.g on smithi098 2024-03-18T20:24:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3456029334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:boot 2024-03-18T20:24:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: fsmap 9 up:standby 2024-03-18T20:24:31.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:31 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T20:24:32.092 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:24:31 smithi098 systemd[1]: Starting Ceph mds.g for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:32.529 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:24:32 smithi098 bash[94415]: 3d72b83cddf3e4d46c621dd3ad631f92bb4930213ada477c9db459a7a867a57c 2024-03-18T20:24:32.530 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:24:32 smithi098 systemd[1]: Started Ceph mds.g for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:32.530 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mds[94453]: -- 172.21.15.98:0/4203974959 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5572c63701a0 con 0x5572c6346c00 2024-03-18T20:24:32.530 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-g[94430]: starting mds.g at 2024-03-18T20:24:32.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 -- 172.21.15.98:0/4035611873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 msgr2=0x7fc48c1090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 --2- 172.21.15.98:0/4035611873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c1090d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc48400b0d0 tx=0x7fc48400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 -- 172.21.15.98:0/4035611873 shutdown_connections 2024-03-18T20:24:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 --2- 172.21.15.98:0/4035611873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c1090d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 --2- 172.21.15.98:0/4035611873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 0x7fc48c1010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 --2- 172.21.15.98:0/4035611873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 0x7fc48c1022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 -- 172.21.15.98:0/4035611873 >> 172.21.15.98:0/4035611873 conn(0x7fc48c0fc200 msgr2=0x7fc48c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 -- 172.21.15.98:0/4035611873 shutdown_connections 2024-03-18T20:24:32.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.741+0000 7fc490ecc700 1 -- 172.21.15.98:0/4035611873 wait complete. 2024-03-18T20:24:32.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 Processor -- start 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 -- start start 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 0x7fc48c196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 0x7fc48c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c19e720 con 0x7fc48c102820 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c19e860 con 0x7fc48c100c90 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc490ecc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c19e9f0 con 0x7fc48c101e80 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc48ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc48ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49790/0 (socket says 172.21.15.98:49790) 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc48ad9d700 1 -- 172.21.15.98:0/993385917 learned_addr learned my addr 172.21.15.98:0/993385917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.742+0000 7fc48a59c700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 0x7fc48c196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc489d9b700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 0x7fc48c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 -- 172.21.15.98:0/993385917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 msgr2=0x7fc48c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 0x7fc48c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 -- 172.21.15.98:0/993385917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 msgr2=0x7fc48c196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 0x7fc48c196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 -- 172.21.15.98:0/993385917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484009d00 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc48ad9d700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc484007b60 tx=0x7fc484012620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48400e040 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc484012dc0 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c19ec70 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484005630 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.743+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c19f140 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.744+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c108040 con 0x7fc48c102820 2024-03-18T20:24:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.745+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc484005790 con 0x7fc48c102820 2024-03-18T20:24:32.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.746+0000 7fc47b7fe700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 0x7fc4740a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:32.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.746+0000 7fc48a59c700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 0x7fc4740a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:32.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.746+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc4840ce640 con 0x7fc48c102820 2024-03-18T20:24:32.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.746+0000 7fc48a59c700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 0x7fc4740a36e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc48c101ce0 tx=0x7fc47c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:32.752+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc484098260 con 0x7fc48c102820 2024-03-18T20:24:32.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:32.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:32.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:32.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:32 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:32.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:32 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:32.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:32 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.012+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc48c067a10 con 0x7fc48c102820 2024-03-18T20:24:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.013+0000 7fc47b7fe700 1 -- 172.21.15.98:0/993385917 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7fc48409b880 con 0x7fc48c102820 2024-03-18T20:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-03-18T20:24:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 msgr2=0x7fc4740a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 0x7fc4740a36e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc48c101ce0 tx=0x7fc47c00a3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 msgr2=0x7fc48c19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc484007b60 tx=0x7fc484012620 comp rx=0 tx=0).stop 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 shutdown_connections 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc48c102820 0x7fc48c19e150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4740a1220 0x7fc4740a36e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc48c100c90 0x7fc48c196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 --2- 172.21.15.98:0/993385917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc48c101e80 0x7fc48c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 >> 172.21.15.98:0/993385917 conn(0x7fc48c0fc200 msgr2=0x7fc48c105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.015+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 shutdown_connections 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:33.016+0000 7fc490ecc700 1 -- 172.21.15.98:0/993385917 wait complete. 2024-03-18T20:24:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:24:33 smithi098 systemd[1]: Starting Ceph mds.a for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mon[43462]: Deploying daemon mds.a on smithi098 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.98:6838/655758721,v1:172.21.15.98:6839/655758721] up:boot 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mon[43462]: fsmap 10 up:standby 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T20:24:33.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/993385917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:33.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:33 smithi169 ceph-mon[48529]: Deploying daemon mds.a on smithi098 2024-03-18T20:24:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:33 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.98:6838/655758721,v1:172.21.15.98:6839/655758721] up:boot 2024-03-18T20:24:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:33 smithi169 ceph-mon[48529]: fsmap 10 up:standby 2024-03-18T20:24:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:33 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T20:24:33.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/993385917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:33.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:33 smithi136 ceph-mon[48073]: Deploying daemon mds.a on smithi098 2024-03-18T20:24:33.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:33 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.98:6838/655758721,v1:172.21.15.98:6839/655758721] up:boot 2024-03-18T20:24:33.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:33 smithi136 ceph-mon[48073]: fsmap 10 up:standby 2024-03-18T20:24:33.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:33 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T20:24:33.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/993385917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:34.342 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-mds[94842]: -- 172.21.15.98:0/587011396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f11d6e1a0 con 0x564f11d44800 2024-03-18T20:24:34.343 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:24:33 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-a[94819]: starting mds.a at 2024-03-18T20:24:34.343 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:24:33 smithi098 bash[94804]: 68a5657ed565e99814256bb73b741894c8dbdf9953f3bcdb401eafcc89150fb5 2024-03-18T20:24:34.343 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:24:33 smithi098 systemd[1]: Started Ceph mds.a for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:34.808 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:34.808 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.808 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.808 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.808 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:34.809 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:34.809 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:34 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:34.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:34 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:34.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T20:24:34.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:34 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:35.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 -- 172.21.15.98:0/1898789041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 msgr2=0x7f30e41011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 --2- 172.21.15.98:0/1898789041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e41011b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30d80099c0 tx=0x7f30d8009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 -- 172.21.15.98:0/1898789041 shutdown_connections 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 --2- 172.21.15.98:0/1898789041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 --2- 172.21.15.98:0/1898789041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 0x7f30e40fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 --2- 172.21.15.98:0/1898789041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e41011b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 -- 172.21.15.98:0/1898789041 >> 172.21.15.98:0/1898789041 conn(0x7f30e40fb0d0 msgr2=0x7f30e40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.169+0000 7f30eb894700 1 -- 172.21.15.98:0/1898789041 shutdown_connections 2024-03-18T20:24:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.170+0000 7f30eb894700 1 -- 172.21.15.98:0/1898789041 wait complete. 2024-03-18T20:24:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.170+0000 7f30eb894700 1 Processor -- start 2024-03-18T20:24:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.170+0000 7f30eb894700 1 -- start start 2024-03-18T20:24:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 0x7f30e4068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e419e570 con 0x7f30e4100d50 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e419e6e0 con 0x7f30e40ffb60 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e419e8a0 con 0x7f30e41016f0 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54406/0 (socket says 172.21.15.98:54406) 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 -- 172.21.15.98:0/1800422782 learned_addr learned my addr 172.21.15.98:0/1800422782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e9630700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 0x7f30e4068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e9e31700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 -- 172.21.15.98:0/1800422782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 msgr2=0x7f30e419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e419ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 -- 172.21.15.98:0/1800422782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 msgr2=0x7f30e4068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 0x7f30e4068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 -- 172.21.15.98:0/1800422782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80095f0 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e9e31700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30e8e2f700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f30d800bf10 tx=0x7f30d800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d801d070 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f30d80054e0 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e419eb20 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8004a90 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.171+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e419f020 con 0x7f30e4100d50 2024-03-18T20:24:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.172+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4106fe0 con 0x7f30e4100d50 2024-03-18T20:24:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.173+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f30d8004c10 con 0x7f30e4100d50 2024-03-18T20:24:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.174+0000 7f30d67fc700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 0x7f30d00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.174+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f30d80cd580 con 0x7f30e4100d50 2024-03-18T20:24:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.174+0000 7f30e9630700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 0x7f30d00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.174+0000 7f30e9630700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 0x7f30d00a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f30e4100bb0 tx=0x7f30e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.177+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f30d8097240 con 0x7f30e4100d50 2024-03-18T20:24:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.421+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30e4067a10 con 0x7f30e4100d50 2024-03-18T20:24:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.422+0000 7f30d67fc700 1 -- 172.21.15.98:0/1800422782 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8846 (secure 0 0 0) 0x7f30d809a860 con 0x7f30e4100d50 2024-03-18T20:24:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:35.426 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-03-18T20:24:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 msgr2=0x7f30d00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 0x7f30d00a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f30e4100bb0 tx=0x7f30e0009260 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 msgr2=0x7f30e419b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f30d800bf10 tx=0x7f30d800fa00 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 shutdown_connections 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30e41016f0 0x7f30e419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f30d00a11a0 0x7f30d00a3660 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e40ffb60 0x7f30e4068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 --2- 172.21.15.98:0/1800422782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e4100d50 0x7f30e419b7e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 >> 172.21.15.98:0/1800422782 conn(0x7f30e40fb0d0 msgr2=0x7f30e4104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 shutdown_connections 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:35.424+0000 7f30eb894700 1 -- 172.21.15.98:0/1800422782 wait complete. 2024-03-18T20:24:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-03-18T20:24:35.491 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:35 smithi136 systemd[1]: Starting Ceph mds.b for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:24:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:35 smithi098 ceph-mon[43462]: Deploying daemon mds.b on smithi136 2024-03-18T20:24:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:35 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.98:6840/2638633847,v1:172.21.15.98:6841/2638633847] up:boot 2024-03-18T20:24:35.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:35 smithi098 ceph-mon[43462]: fsmap 11 up:standby 2024-03-18T20:24:35.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:35 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T20:24:35.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:35 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1800422782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:35 smithi169 ceph-mon[48529]: Deploying daemon mds.b on smithi136 2024-03-18T20:24:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:35 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.98:6840/2638633847,v1:172.21.15.98:6841/2638633847] up:boot 2024-03-18T20:24:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:35 smithi169 ceph-mon[48529]: fsmap 11 up:standby 2024-03-18T20:24:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:35 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T20:24:35.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:35 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1800422782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mon[48073]: Deploying daemon mds.b on smithi136 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.98:6840/2638633847,v1:172.21.15.98:6841/2638633847] up:boot 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mon[48073]: fsmap 11 up:standby 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1800422782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-mds[84692]: -- 172.21.15.136:0/2914174035 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55984a2381a0 con 0x55984a20e800 2024-03-18T20:24:35.991 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:35 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-b[84669]: starting mds.b at 2024-03-18T20:24:35.992 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:35 smithi136 bash[84654]: fae059698b04fef2c2ab01824151c2629305ffd4c2deed2295ce17f59dd5a2b5 2024-03-18T20:24:35.992 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:35 smithi136 systemd[1]: Started Ceph mds.b for a6e2b072-e564-11ee-95c9-87774f69a715. 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:boot 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: fsmap 12 up:standby 2024-03-18T20:24:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:36 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:boot 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: fsmap 12 up:standby 2024-03-18T20:24:37.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:36 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T20:24:37.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:37.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:boot 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: fsmap 12 up:standby 2024-03-18T20:24:37.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:36 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T20:24:37.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 -- 172.21.15.98:0/4291117296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 msgr2=0x7f27fc109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 --2- 172.21.15.98:0/4291117296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc109150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f27f400b0a0 tx=0x7f27f400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 -- 172.21.15.98:0/4291117296 shutdown_connections 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 --2- 172.21.15.98:0/4291117296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc109150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 --2- 172.21.15.98:0/4291117296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc0fef30 0x7f27fc0ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 --2- 172.21.15.98:0/4291117296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc101f00 0x7f27fc102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 -- 172.21.15.98:0/4291117296 >> 172.21.15.98:0/4291117296 conn(0x7f27fc0fa880 msgr2=0x7f27fc0fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 -- 172.21.15.98:0/4291117296 shutdown_connections 2024-03-18T20:24:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.577+0000 7f2801ebf700 1 -- 172.21.15.98:0/4291117296 wait complete. 2024-03-18T20:24:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 Processor -- start 2024-03-18T20:24:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 -- start start 2024-03-18T20:24:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 0x7f27fc101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc101f00 0x7f27fc0ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc100b90 con 0x7f27fc1028a0 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc100d00 con 0x7f27fc0fef30 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f2801ebf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc1a0cb0 con 0x7f27fc101f00 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fb7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 0x7f27fc101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54432/0 (socket says 172.21.15.98:54432) 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 -- 172.21.15.98:0/1474633929 learned_addr learned my addr 172.21.15.98:0/1474633929 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27faffd700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc101f00 0x7f27fc0ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 -- 172.21.15.98:0/1474633929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc101f00 msgr2=0x7f27fc0ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc101f00 0x7f27fc0ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 -- 172.21.15.98:0/1474633929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 msgr2=0x7f27fc101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 0x7f27fc101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fbfff700 1 -- 172.21.15.98:0/1474633929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4009d00 con 0x7f27fc1028a0 2024-03-18T20:24:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.578+0000 7f27fb7fe700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 0x7f27fc101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f27fbfff700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f27f4007a30 tx=0x7f27f4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f400e040 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27f40056e0 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc1a0ee0 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f401dad0 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.579+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc1a12a0 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.580+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc1081e0 con 0x7f27fc1028a0 2024-03-18T20:24:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.581+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f27f4005850 con 0x7f27fc1028a0 2024-03-18T20:24:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.581+0000 7f27f8ff9700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 0x7f27e40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.581+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f27f40cdd10 con 0x7f27fc1028a0 2024-03-18T20:24:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.581+0000 7f27fb7fe700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 0x7f27e40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.582+0000 7f27fb7fe700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 0x7f27e40a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27fc067bf0 tx=0x7f27ec00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.586+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f27f4097930 con 0x7f27fc1028a0 2024-03-18T20:24:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.845+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27fc02d130 con 0x7f27fc1028a0 2024-03-18T20:24:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.845+0000 7f27f8ff9700 1 -- 172.21.15.98:0/1474633929 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7f27fc02d130 con 0x7f27fc1028a0 2024-03-18T20:24:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:37.849 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-03-18T20:24:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 msgr2=0x7f27e40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 0x7f27e40a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27fc067bf0 tx=0x7f27ec00a380 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 msgr2=0x7f27fc1003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f27f4007a30 tx=0x7f27f4004c80 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 shutdown_connections 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1028a0 0x7f27fc1003f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f27e40a11a0 0x7f27e40a3660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27fc0fef30 0x7f27fc101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 --2- 172.21.15.98:0/1474633929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27fc101f00 0x7f27fc0ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 >> 172.21.15.98:0/1474633929 conn(0x7f27fc0fa880 msgr2=0x7f27fc105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 shutdown_connections 2024-03-18T20:24:37.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.848+0000 7f2801ebf700 1 -- 172.21.15.98:0/1474633929 wait complete. 2024-03-18T20:24:37.853 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-03-18T20:24:37.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 -- 172.21.15.98:0/1991909820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda780ffec0 msgr2=0x7fda78100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/1991909820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda780ffec0 0x7fda78100340 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0099c0 tx=0x7fda6c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 -- 172.21.15.98:0/1991909820 shutdown_connections 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/1991909820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda78100880 0x7fda78100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/1991909820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 0x7fda780ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/1991909820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda780ffec0 0x7fda78100340 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 -- 172.21.15.98:0/1991909820 >> 172.21.15.98:0/1991909820 conn(0x7fda780fb100 msgr2=0x7fda780fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 -- 172.21.15.98:0/1991909820 shutdown_connections 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.993+0000 7fda7dcd6700 1 -- 172.21.15.98:0/1991909820 wait complete. 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 Processor -- start 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 -- start start 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 0x7fda78198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda780ffec0 0x7fda78199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a0790 con 0x7fda78100880 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a0900 con 0x7fda780ffec0 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda7dcd6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a0ac0 con 0x7fda780ff560 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda77fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda77fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54444/0 (socket says 172.21.15.98:54444) 2024-03-18T20:24:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda77fff700 1 -- 172.21.15.98:0/2223318566 learned_addr learned my addr 172.21.15.98:0/2223318566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda777fe700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 0x7fda78198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.994+0000 7fda76ffd700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda780ffec0 0x7fda78199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 -- 172.21.15.98:0/2223318566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 msgr2=0x7fda78198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 0x7fda78198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 -- 172.21.15.98:0/2223318566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda780ffec0 msgr2=0x7fda78199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda780ffec0 0x7fda78199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 -- 172.21.15.98:0/2223318566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0095f0 con 0x7fda78100880 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda77fff700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda7000ef30 tx=0x7fda7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70018070 con 0x7fda78100880 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda781a0d40 con 0x7fda78100880 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fda7000f040 con 0x7fda78100880 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000cbc0 con 0x7fda78100880 2024-03-18T20:24:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.995+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda781a11a0 con 0x7fda78100880 2024-03-18T20:24:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.996+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7810a340 con 0x7fda78100880 2024-03-18T20:24:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.997+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fda7000cd80 con 0x7fda78100880 2024-03-18T20:24:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.998+0000 7fda74ff9700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 0x7fda600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.998+0000 7fda777fe700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 0x7fda600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.998+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda700cbf30 con 0x7fda78100880 2024-03-18T20:24:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:37.998+0000 7fda777fe700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 0x7fda600a35e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda7819a120 tx=0x7fda68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.003+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda70095bf0 con 0x7fda78100880 2024-03-18T20:24:38.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.223+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda7802ce40 con 0x7fda78100880 2024-03-18T20:24:38.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.224+0000 7fda74ff9700 1 -- 172.21.15.98:0/2223318566 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18422 (secure 0 0 0) 0x7fda70099210 con 0x7fda78100880 2024-03-18T20:24:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:38.228 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":77,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:09.636901+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:38.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 msgr2=0x7fda600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 0x7fda600a35e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda7819a120 tx=0x7fda68009260 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 msgr2=0x7fda781a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda7000ef30 tx=0x7fda7000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 shutdown_connections 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda78100880 0x7fda781a01c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda780ff560 0x7fda78198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fda600a1120 0x7fda600a35e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 --2- 172.21.15.98:0/2223318566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda780ffec0 0x7fda78199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 >> 172.21.15.98:0/2223318566 conn(0x7fda780fb100 msgr2=0x7fda780fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 shutdown_connections 2024-03-18T20:24:38.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.227+0000 7fda7dcd6700 1 -- 172.21.15.98:0/2223318566 wait complete. 2024-03-18T20:24:38.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:37 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1474633929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:37 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1474633929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:24:38.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 -- 172.21.15.98:0/635197113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d80ff030 msgr2=0x7fe2d80ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/635197113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d80ff030 0x7fe2d80ff450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009990 tx=0x7fe2d4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 -- 172.21.15.98:0/635197113 shutdown_connections 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/635197113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d81029a0 0x7fe2d8109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/635197113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d80ff030 0x7fe2d80ff450 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/635197113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d8102000 0x7fe2d8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 -- 172.21.15.98:0/635197113 >> 172.21.15.98:0/635197113 conn(0x7fe2d80fa960 msgr2=0x7fe2d80fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 -- 172.21.15.98:0/635197113 shutdown_connections 2024-03-18T20:24:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 -- 172.21.15.98:0/635197113 wait complete. 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.373+0000 7fe2dfebf700 1 Processor -- start 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 -- start start 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d80ff030 0x7fe2d8101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d8102000 0x7fe2d80fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d8100c90 con 0x7fe2d81029a0 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d8100e00 con 0x7fe2d8102000 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dfebf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d81a0d20 con 0x7fe2d80ff030 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54462/0 (socket says 172.21.15.98:54462) 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 -- 172.21.15.98:0/2532358662 learned_addr learned my addr 172.21.15.98:0/2532358662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2dd45a700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d8102000 0x7fe2d80fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2ddc5b700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d80ff030 0x7fe2d8101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 -- 172.21.15.98:0/2532358662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d80ff030 msgr2=0x7fe2d8101920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d80ff030 0x7fe2d8101920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 -- 172.21.15.98:0/2532358662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d8102000 msgr2=0x7fe2d80fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d8102000 0x7fe2d80fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 -- 172.21.15.98:0/2532358662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d40095f0 con 0x7fe2d81029a0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2de45c700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000ef30 tx=0x7fe2d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.374+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0018070 con 0x7fe2d81029a0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.375+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe2d000f040 con 0x7fe2d81029a0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.375+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d81a0f50 con 0x7fe2d81029a0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.375+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000cb60 con 0x7fe2d81029a0 2024-03-18T20:24:38.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.375+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d81a1380 con 0x7fe2d81029a0 2024-03-18T20:24:38.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.375+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d8108380 con 0x7fe2d81029a0 2024-03-18T20:24:38.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.376+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe2d0007500 con 0x7fe2d81029a0 2024-03-18T20:24:38.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.377+0000 7fe2ceffd700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 0x7fe2c40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.377+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe2d00cbf10 con 0x7fe2d81029a0 2024-03-18T20:24:38.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.377+0000 7fe2ddc5b700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 0x7fe2c40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.378+0000 7fe2ddc5b700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 0x7fe2c40a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4005910 tx=0x7fe2d400b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:37 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1474633929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.382+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe2d0095bb0 con 0x7fe2d81029a0 2024-03-18T20:24:38.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.608+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2d802ccb0 con 0x7fe2d81029a0 2024-03-18T20:24:38.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.608+0000 7fe2ceffd700 1 -- 172.21.15.98:0/2532358662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2d00991d0 con 0x7fe2d81029a0 2024-03-18T20:24:38.612 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.610+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 msgr2=0x7fe2c40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.610+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 0x7fe2c40a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4005910 tx=0x7fe2d400b610 comp rx=0 tx=0).stop 2024-03-18T20:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.610+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 msgr2=0x7fe2d81004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.610+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000ef30 tx=0x7fe2d000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 shutdown_connections 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2d81029a0 0x7fe2d81004f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2d80ff030 0x7fe2d8101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2c40a1120 0x7fe2c40a35e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 --2- 172.21.15.98:0/2532358662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d8102000 0x7fe2d80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 >> 172.21.15.98:0/2532358662 conn(0x7fe2d80fa960 msgr2=0x7fe2d81054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 shutdown_connections 2024-03-18T20:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.611+0000 7fe2dfebf700 1 -- 172.21.15.98:0/2532358662 wait complete. 2024-03-18T20:24:38.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 -- 172.21.15.98:0/307356912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b440fef30 msgr2=0x7f5b440ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 --2- 172.21.15.98:0/307356912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b440fef30 0x7f5b440ff350 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009990 tx=0x7f5b38009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 -- 172.21.15.98:0/307356912 shutdown_connections 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 --2- 172.21.15.98:0/307356912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b441028a0 0x7f5b44109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 --2- 172.21.15.98:0/307356912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b440fef30 0x7f5b440ff350 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 --2- 172.21.15.98:0/307356912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b44102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 -- 172.21.15.98:0/307356912 >> 172.21.15.98:0/307356912 conn(0x7f5b440fa880 msgr2=0x7f5b440fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 -- 172.21.15.98:0/307356912 shutdown_connections 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 -- 172.21.15.98:0/307356912 wait complete. 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.763+0000 7f5b4b020700 1 Processor -- start 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 -- start start 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b440fef30 0x7f5b44196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b441970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4419e690 con 0x7f5b441028a0 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4419e7d0 con 0x7f5b44101f00 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b4b020700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4419e960 con 0x7f5b440fef30 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54466/0 (socket says 172.21.15.98:54466) 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 -- 172.21.15.98:0/3694063957 learned_addr learned my addr 172.21.15.98:0/3694063957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b48dbc700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b440fef30 0x7f5b44196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b43fff700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b441970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 -- 172.21.15.98:0/3694063957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b440fef30 msgr2=0x7f5b44196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b440fef30 0x7f5b44196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 -- 172.21.15.98:0/3694063957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 msgr2=0x7f5b441970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b441970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 -- 172.21.15.98:0/3694063957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30009520 con 0x7f5b441028a0 2024-03-18T20:24:38.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b43fff700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b441970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:24:38.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.764+0000 7f5b495bd700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00ef30 tx=0x7f5b3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.766+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c018070 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.766+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5b3c00f040 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.766+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b380095f0 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.766+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00cbc0 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.766+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4419ef60 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.767+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b44108220 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.768+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f5b3c007500 con 0x7f5b441028a0 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.768+0000 7f5b41ffb700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 0x7f5b340a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:38.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.768+0000 7f5b48dbc700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 0x7f5b340a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:38.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.769+0000 7f5b48dbc700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 0x7f5b340a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b38005910 tx=0x7f5b3800b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:38.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.770+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5b3c012340 con 0x7f5b441028a0 2024-03-18T20:24:38.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.775+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5b3c095b60 con 0x7f5b441028a0 2024-03-18T20:24:39.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.998+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5b440619c0 con 0x7f5b441028a0 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:38.998+0000 7f5b41ffb700 1 -- 172.21.15.98:0/3694063957 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5b3c099180 con 0x7f5b441028a0 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:24:39.002 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 msgr2=0x7f5b340a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 0x7f5b340a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b38005910 tx=0x7f5b3800b610 comp rx=0 tx=0).stop 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 msgr2=0x7f5b4419e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00ef30 tx=0x7f5b3c00c3a0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 shutdown_connections 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b441028a0 0x7f5b4419e0c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5b440fef30 0x7f5b44196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b340a11a0 0x7f5b340a3660 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 --2- 172.21.15.98:0/3694063957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b44101f00 0x7f5b441970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 >> 172.21.15.98:0/3694063957 conn(0x7f5b440fa880 msgr2=0x7f5b44105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 shutdown_connections 2024-03-18T20:24:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.000+0000 7f5b4b020700 1 -- 172.21.15.98:0/3694063957 wait complete. 2024-03-18T20:24:39.016 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:24:39.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:24:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 -- 172.21.15.98:0/293966559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 msgr2=0x7f7758103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 --2- 172.21.15.98:0/293966559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758103bd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f774c009990 tx=0x7f774c009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 -- 172.21.15.98:0/293966559 shutdown_connections 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 --2- 172.21.15.98:0/293966559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758106a40 0x7f7758108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 --2- 172.21.15.98:0/293966559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758103bd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 --2- 172.21.15.98:0/293966559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758104110 0x7f7758106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 -- 172.21.15.98:0/293966559 >> 172.21.15.98:0/293966559 conn(0x7f77580fb1d0 msgr2=0x7f77580fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 -- 172.21.15.98:0/293966559 shutdown_connections 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.138+0000 7f775f97c700 1 -- 172.21.15.98:0/293966559 wait complete. 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 Processor -- start 2024-03-18T20:24:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 -- start start 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758104110 0x7f7758197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758106a40 0x7f775819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775819e760 con 0x7f77581017e0 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775819e8a0 con 0x7f7758104110 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775f97c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775819ea30 con 0x7f7758106a40 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54476/0 (socket says 172.21.15.98:54476) 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 -- 172.21.15.98:0/4234208595 learned_addr learned my addr 172.21.15.98:0/4234208595 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775cf17700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758104110 0x7f7758197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775df19700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758106a40 0x7f775819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 -- 172.21.15.98:0/4234208595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758106a40 msgr2=0x7f775819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758106a40 0x7f775819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 -- 172.21.15.98:0/4234208595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758104110 msgr2=0x7f7758197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758104110 0x7f7758197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 -- 172.21.15.98:0/4234208595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c0095f0 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.139+0000 7f775d718700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f774c00b2e0 tx=0x7f774c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.140+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c01d070 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.140+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f774c00beb0 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.140+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775819ecb0 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.140+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c021410 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.140+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775819f130 con 0x7f77581017e0 2024-03-18T20:24:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.141+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77580fcd50 con 0x7f77581017e0 2024-03-18T20:24:39.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.142+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f774c00f640 con 0x7f77581017e0 2024-03-18T20:24:39.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.142+0000 7f774a7fc700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 0x7f77440a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:39.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.142+0000 7f775cf17700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 0x7f77440a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.142+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f774c0cdb60 con 0x7f77581017e0 2024-03-18T20:24:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.142+0000 7f775cf17700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 0x7f77440a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7754005950 tx=0x7f775400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.145+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f774c097820 con 0x7f77581017e0 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2223318566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2532358662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:39.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:38 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2223318566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2532358662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:39.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:38 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.344+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7758067ac0 con 0x7f77581017e0 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2223318566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2532358662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:24:39.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:38 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.926+0000 7f774a7fc700 1 -- 172.21.15.98:0/4234208595 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f774c09ae40 con 0x7f77581017e0 2024-03-18T20:24:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.930+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 msgr2=0x7f77440a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 0x7f77440a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7754005950 tx=0x7f775400a300 comp rx=0 tx=0).stop 2024-03-18T20:24:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 msgr2=0x7f7758196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f774c00b2e0 tx=0x7f774c004730 comp rx=0 tx=0).stop 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 shutdown_connections 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7758106a40 0x7f775819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77581017e0 0x7f7758196c30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77440a11a0 0x7f77440a3660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 --2- 172.21.15.98:0/4234208595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7758104110 0x7f7758197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 >> 172.21.15.98:0/4234208595 conn(0x7f77580fb1d0 msgr2=0x7f7758103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 shutdown_connections 2024-03-18T20:24:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:39.931+0000 7f775f97c700 1 -- 172.21.15.98:0/4234208595 wait complete. 2024-03-18T20:24:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-03-18T20:24:39.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T20:24:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 -- 172.21.15.98:0/242240119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f40ffc60 msgr2=0x7f98f4100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 --2- 172.21.15.98:0/242240119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f40ffc60 0x7f98f4100080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f98e40099c0 tx=0x7f98e4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 -- 172.21.15.98:0/242240119 shutdown_connections 2024-03-18T20:24:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 --2- 172.21.15.98:0/242240119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f41017f0 0x7f98f41080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 --2- 172.21.15.98:0/242240119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f40ffc60 0x7f98f4100080 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 --2- 172.21.15.98:0/242240119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f4100e50 0x7f98f41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 -- 172.21.15.98:0/242240119 >> 172.21.15.98:0/242240119 conn(0x7f98f40fb1d0 msgr2=0x7f98f40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 -- 172.21.15.98:0/242240119 shutdown_connections 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.073+0000 7f98f895b700 1 -- 172.21.15.98:0/242240119 wait complete. 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 Processor -- start 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 -- start start 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f40ffc60 0x7f98f4196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f4100e50 0x7f98f41971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419e770 con 0x7f98f41017f0 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419e8e0 con 0x7f98f4100e50 2024-03-18T20:24:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f895b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419eaa0 con 0x7f98f40ffc60 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54508/0 (socket says 172.21.15.98:54508) 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.074+0000 7f98f2d9d700 1 -- 172.21.15.98:0/832217730 learned_addr learned my addr 172.21.15.98:0/832217730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f259c700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f40ffc60 0x7f98f4196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f1d9b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f4100e50 0x7f98f41971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 -- 172.21.15.98:0/832217730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f40ffc60 msgr2=0x7f98f4196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f40ffc60 0x7f98f4196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 -- 172.21.15.98:0/832217730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f4100e50 msgr2=0x7f98f41971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f4100e50 0x7f98f41971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 -- 172.21.15.98:0/832217730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40095f0 con 0x7f98f41017f0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f2d9d700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ada0 tx=0x7f98ec00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00f070 con 0x7f98f41017f0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98ec007b30 con 0x7f98f41017f0 2024-03-18T20:24:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec005280 con 0x7f98f41017f0 2024-03-18T20:24:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f419ed80 con 0x7f98f41017f0 2024-03-18T20:24:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.075+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f419f1b0 con 0x7f98f41017f0 2024-03-18T20:24:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.076+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f4067af0 con 0x7f98f41017f0 2024-03-18T20:24:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.077+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f98ec00a090 con 0x7f98f41017f0 2024-03-18T20:24:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.078+0000 7f98e37fe700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 0x7f98dc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.078+0000 7f98f259c700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 0x7f98dc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.078+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f98ec0cc090 con 0x7f98f41017f0 2024-03-18T20:24:40.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.078+0000 7f98f259c700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 0x7f98dc0a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98e4006010 tx=0x7f98e4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.081+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98ec095be0 con 0x7f98f41017f0 2024-03-18T20:24:40.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:39 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3694063957' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T20:24:40.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:39 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T20:24:40.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:39 smithi136 ceph-mon[48073]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:40.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.278+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f98f419f340 con 0x7f98f41017f0 2024-03-18T20:24:40.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:39 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3694063957' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T20:24:40.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:39 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T20:24:40.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:39 smithi098 ceph-mon[43462]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:40.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:39 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3694063957' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T20:24:40.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:39 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T20:24:40.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:39 smithi169 ceph-mon[48529]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T20:24:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.926+0000 7f98e37fe700 1 -- 172.21.15.98:0/832217730 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v79) v1 ==== 171+0+0 (secure 0 0 0) 0x7f98ec099200 con 0x7f98f41017f0 2024-03-18T20:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.932+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 msgr2=0x7f98dc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.932+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 0x7f98dc0a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98e4006010 tx=0x7f98e4000bc0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.932+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 msgr2=0x7f98f419e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.932+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ada0 tx=0x7f98ec00c330 comp rx=0 tx=0).stop 2024-03-18T20:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 shutdown_connections 2024-03-18T20:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98f41017f0 0x7f98f419e1a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98f40ffc60 0x7f98f4196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98dc0a11a0 0x7f98dc0a3660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 --2- 172.21.15.98:0/832217730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98f4100e50 0x7f98f41971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.936+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 >> 172.21.15.98:0/832217730 conn(0x7f98f40fb1d0 msgr2=0x7f98f4067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.937+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 shutdown_connections 2024-03-18T20:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:40.937+0000 7f98f895b700 1 -- 172.21.15.98:0/832217730 wait complete. 2024-03-18T20:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-03-18T20:24:40.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 -- 172.21.15.98:0/1996313168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e0101290 msgr2=0x7f00e01016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 --2- 172.21.15.98:0/1996313168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e0101290 0x7f00e01016b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009990 tx=0x7f00d0009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 -- 172.21.15.98:0/1996313168 shutdown_connections 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 --2- 172.21.15.98:0/1996313168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 0x7f00e0108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 --2- 172.21.15.98:0/1996313168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e0101290 0x7f00e01016b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 --2- 172.21.15.98:0/1996313168 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e00ffee0 0x7f00e0100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 -- 172.21.15.98:0/1996313168 >> 172.21.15.98:0/1996313168 conn(0x7f00e00fa990 msgr2=0x7f00e00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 -- 172.21.15.98:0/1996313168 shutdown_connections 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.083+0000 7f00e53cf700 1 -- 172.21.15.98:0/1996313168 wait complete. 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 Processor -- start 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 -- start start 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 0x7f00e0197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e0101290 0x7f00e019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e019e7e0 con 0x7f00e00ffee0 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e019e950 con 0x7f00e01008a0 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00e53cf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e019eb10 con 0x7f00e0101290 2024-03-18T20:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54514/0 (socket says 172.21.15.98:54514) 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 -- 172.21.15.98:0/4241700185 learned_addr learned my addr 172.21.15.98:0/4241700185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00df7fe700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e0101290 0x7f00e019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00de7fc700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 0x7f00e0197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 -- 172.21.15.98:0/4241700185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e0101290 msgr2=0x7f00e019e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e0101290 0x7f00e019e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 -- 172.21.15.98:0/4241700185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 msgr2=0x7f00e0197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.084+0000 7f00deffd700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 0x7f00e0197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00deffd700 1 -- 172.21.15.98:0/4241700185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d00095f0 con 0x7f00e00ffee0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00deffd700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f00d00098e0 tx=0x7f00d0005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d001d070 con 0x7f00e00ffee0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e019ed90 con 0x7f00e00ffee0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00d0004000 con 0x7f00e00ffee0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0022ca0 con 0x7f00e00ffee0 2024-03-18T20:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.085+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e019f1c0 con 0x7f00e00ffee0 2024-03-18T20:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.086+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e0067f40 con 0x7f00e00ffee0 2024-03-18T20:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.088+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f00d0022470 con 0x7f00e00ffee0 2024-03-18T20:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.088+0000 7f00d7fff700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 0x7f00cc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.088+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f00d00ce2d0 con 0x7f00e00ffee0 2024-03-18T20:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.088+0000 7f00de7fc700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 0x7f00cc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.089+0000 7f00de7fc700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 0x7f00cc0a35e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00c8005950 tx=0x7f00c80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.092+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f00d0097ca0 con 0x7f00e00ffee0 2024-03-18T20:24:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:40 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T20:24:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:40 smithi136 ceph-mon[48073]: osdmap e78: 12 total, 12 up, 12 in 2024-03-18T20:24:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:40 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T20:24:41.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:41.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.334+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f00e019f350 con 0x7f00e00ffee0 2024-03-18T20:24:41.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T20:24:41.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:40 smithi098 ceph-mon[43462]: osdmap e78: 12 total, 12 up, 12 in 2024-03-18T20:24:41.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T20:24:41.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:41.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.345+0000 7f00d7fff700 1 -- 172.21.15.98:0/4241700185 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f00d009b2c0 con 0x7f00e00ffee0 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.355+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 msgr2=0x7f00cc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.355+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 0x7f00cc0a35e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00c8005950 tx=0x7f00c80058e0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.355+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 msgr2=0x7f00e0196cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.355+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f00d00098e0 tx=0x7f00d0005f10 comp rx=0 tx=0).stop 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 shutdown_connections 2024-03-18T20:24:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f00e0101290 0x7f00e019e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00e00ffee0 0x7f00e0196cc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f00cc0a1120 0x7f00cc0a35e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 --2- 172.21.15.98:0/4241700185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00e01008a0 0x7f00e0197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.357+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 >> 172.21.15.98:0/4241700185 conn(0x7f00e00fa990 msgr2=0x7f00e00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.359+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 shutdown_connections 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.359+0000 7f00e53cf700 1 -- 172.21.15.98:0/4241700185 wait complete. 2024-03-18T20:24:41.364 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T20:24:41.379 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T20:24:41.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T20:24:41.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:40 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4234208595' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T20:24:41.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:40 smithi169 ceph-mon[48529]: osdmap e78: 12 total, 12 up, 12 in 2024-03-18T20:24:41.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:40 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T20:24:41.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 -- 172.21.15.98:0/4179976020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 msgr2=0x7f25b0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 --2- 172.21.15.98:0/4179976020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b0106500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009990 tx=0x7f25a0009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 -- 172.21.15.98:0/4179976020 shutdown_connections 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 --2- 172.21.15.98:0/4179976020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b0106a40 0x7f25b0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 --2- 172.21.15.98:0/4179976020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b01017e0 0x7f25b0103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 --2- 172.21.15.98:0/4179976020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b0106500 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 -- 172.21.15.98:0/4179976020 >> 172.21.15.98:0/4179976020 conn(0x7f25b00fb1d0 msgr2=0x7f25b00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 -- 172.21.15.98:0/4179976020 shutdown_connections 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.505+0000 7f25b6891700 1 -- 172.21.15.98:0/4179976020 wait complete. 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 Processor -- start 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 -- start start 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 0x7f25b0101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 0x7f25b00ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b0100470 con 0x7f25b0104110 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b01005e0 con 0x7f25b01017e0 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b6891700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b01a0d70 con 0x7f25b0106a40 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54520/0 (socket says 172.21.15.98:54520) 2024-03-18T20:24:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 -- 172.21.15.98:0/2298613677 learned_addr learned my addr 172.21.15.98:0/2298613677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b4e2e700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 0x7f25b00ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25affff700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 0x7f25b0101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 -- 172.21.15.98:0/2298613677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 msgr2=0x7f25b00ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 0x7f25b00ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 -- 172.21.15.98:0/2298613677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 msgr2=0x7f25b0101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 0x7f25b0101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25af7fe700 1 -- 172.21.15.98:0/2298613677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a00095f0 con 0x7f25b0104110 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25b4e2e700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 0x7f25b00ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.506+0000 7f25affff700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 0x7f25b0101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25af7fe700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f25a0006010 tx=0x7f25a0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a001d070 con 0x7f25b0104110 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f25a0004140 con 0x7f25b0104110 2024-03-18T20:24:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b01a0fa0 con 0x7f25b0104110 2024-03-18T20:24:41.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0022de0 con 0x7f25b0104110 2024-03-18T20:24:41.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.507+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b01a1310 con 0x7f25b0104110 2024-03-18T20:24:41.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.508+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b00fcd50 con 0x7f25b0104110 2024-03-18T20:24:41.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.509+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f25a0022570 con 0x7f25b0104110 2024-03-18T20:24:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.509+0000 7f25ad7fa700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 0x7f25980a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.510+0000 7f25affff700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 0x7f25980a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.510+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f25a00ce3f0 con 0x7f25b0104110 2024-03-18T20:24:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.510+0000 7f25affff700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 0x7f25980a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f25a4007900 tx=0x7f25a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.513+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f25a0097cc0 con 0x7f25b0104110 2024-03-18T20:24:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:41.721+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f25b0067ac0 con 0x7f25b0104110 2024-03-18T20:24:41.948 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-b[84669]: 2024-03-18T20:24:41.547+0000 7f98b7ecc700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: osdmap e79: 12 total, 12 up, 12 in 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T20:24:42.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:active 2024-03-18T20:24:42.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:42.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: pgmap v218: 97 pgs: 15 creating+peering, 71 unknown, 11 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-18T20:24:42.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T20:24:42.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: osdmap e79: 12 total, 12 up, 12 in 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:active 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: pgmap v218: 97 pgs: 15 creating+peering, 71 unknown, 11 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-18T20:24:42.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T20:24:42.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.341+0000 7f25ad7fa700 1 -- 172.21.15.98:0/2298613677 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) v1 ==== 157+0+0 (secure 0 0 0) 0x7f25a00270e0 con 0x7f25b0104110 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 msgr2=0x7f25980a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 0x7f25980a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f25a4007900 tx=0x7f25a4008040 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 msgr2=0x7f25b00ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f25a0006010 tx=0x7f25a0005740 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 shutdown_connections 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25b0106a40 0x7f25b00ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25980a1120 0x7f25980a35e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25b01017e0 0x7f25b0101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 --2- 172.21.15.98:0/2298613677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b0104110 0x7f25b00ff790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 >> 172.21.15.98:0/2298613677 conn(0x7f25b00fb1d0 msgr2=0x7f25b0102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 shutdown_connections 2024-03-18T20:24:42.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.344+0000 7f25b6891700 1 -- 172.21.15.98:0/2298613677 wait complete. 2024-03-18T20:24:42.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:24:42.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/832217730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: osdmap e79: 12 total, 12 up, 12 in 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4241700185' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] up:active 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: pgmap v218: 97 pgs: 15 creating+peering, 71 unknown, 11 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-18T20:24:42.384 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T20:24:42.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 -- 172.21.15.98:0/1988593497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224101e90 msgr2=0x7f22241022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:42.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 --2- 172.21.15.98:0/1988593497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224101e90 0x7f22241022f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f22140099c0 tx=0x7f2214009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 -- 172.21.15.98:0/1988593497 shutdown_connections 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 --2- 172.21.15.98:0/1988593497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f22241090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 --2- 172.21.15.98:0/1988593497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224100ca0 0x7f22241010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 --2- 172.21.15.98:0/1988593497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224101e90 0x7f22241022f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.495+0000 7f22299a0700 1 -- 172.21.15.98:0/1988593497 >> 172.21.15.98:0/1988593497 conn(0x7f22240fc1f0 msgr2=0x7f22240fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- 172.21.15.98:0/1988593497 shutdown_connections 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- 172.21.15.98:0/1988593497 wait complete. 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 Processor -- start 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- start start 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 0x7f22241970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f222419e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222419e6e0 con 0x7f2224100ca0 2024-03-18T20:24:42.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222419e820 con 0x7f2224102830 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f22299a0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222419e9b0 con 0x7f2224101e90 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f2222ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f2222ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54538/0 (socket says 172.21.15.98:54538) 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.496+0000 7f2222ffd700 1 -- 172.21.15.98:0/3134423370 learned_addr learned my addr 172.21.15.98:0/3134423370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22227fc700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 0x7f22241970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 -- 172.21.15.98:0/3134423370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 msgr2=0x7f22241970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 0x7f22241970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 -- 172.21.15.98:0/3134423370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 msgr2=0x7f222419e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22237fe700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f222419e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f222419e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 -- 172.21.15.98:0/3134423370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140095f0 con 0x7f2224100ca0 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22227fc700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 0x7f22241970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22237fe700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f222419e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:42.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f2222ffd700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f221800b500 tx=0x7f221800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218011830 con 0x7f2224100ca0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2218011e20 con 0x7f2224100ca0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222419ec90 con 0x7f2224100ca0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218010dd0 con 0x7f2224100ca0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.497+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222419f1a0 con 0x7f2224100ca0 2024-03-18T20:24:42.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.498+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2224108170 con 0x7f2224100ca0 2024-03-18T20:24:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.499+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f2218011990 con 0x7f2224100ca0 2024-03-18T20:24:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.500+0000 7f222899e700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 0x7f220c0a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.500+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f22180cc7e0 con 0x7f2224100ca0 2024-03-18T20:24:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.500+0000 7f22227fc700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 0x7f220c0a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.500+0000 7f22227fc700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 0x7f220c0a35d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2214003780 tx=0x7f2214009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:42.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.503+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2218095f90 con 0x7f2224100ca0 2024-03-18T20:24:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:42.698+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2224067e60 con 0x7f2224100ca0 2024-03-18T20:24:43.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:43 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T20:24:43.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:43 smithi169 ceph-mon[48529]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T20:24:43.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:43 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T20:24:43.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:43 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T20:24:43.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:43 smithi136 ceph-mon[48073]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T20:24:43.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:43 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T20:24:43.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:43 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2298613677' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T20:24:43.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:43 smithi098 ceph-mon[43462]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T20:24:43.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:43 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T20:24:44.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.428+0000 7f222899e700 1 -- 172.21.15.98:0/3134423370 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) v1 ==== 228+0+0 (secure 0 0 0) 0x7f22180995b0 con 0x7f2224100ca0 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 msgr2=0x7f220c0a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 0x7f220c0a35d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2214003780 tx=0x7f2214009f10 comp rx=0 tx=0).stop 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 msgr2=0x7f2224196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f221800b500 tx=0x7f221800b8c0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 shutdown_connections 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f220c0a1110 0x7f220c0a35d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2224102830 0x7f222419e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2224100ca0 0x7f2224196bb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 --2- 172.21.15.98:0/3134423370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2224101e90 0x7f22241970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 >> 172.21.15.98:0/3134423370 conn(0x7f22240fc1f0 msgr2=0x7f22241052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:44.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 shutdown_connections 2024-03-18T20:24:44.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.435+0000 7f22299a0700 1 -- 172.21.15.98:0/3134423370 wait complete. 2024-03-18T20:24:44.440 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' created 2024-03-18T20:24:44.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 -- 172.21.15.98:0/4189277677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76480ffc60 msgr2=0x7f7648100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 --2- 172.21.15.98:0/4189277677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76480ffc60 0x7f7648100080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f763800b0d0 tx=0x7f763800b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 -- 172.21.15.98:0/4189277677 shutdown_connections 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 --2- 172.21.15.98:0/4189277677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76481017f0 0x7f76481080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 --2- 172.21.15.98:0/4189277677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76480ffc60 0x7f7648100080 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 --2- 172.21.15.98:0/4189277677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 0x7f76481012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 -- 172.21.15.98:0/4189277677 >> 172.21.15.98:0/4189277677 conn(0x7f76480fb1d0 msgr2=0x7f76480fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 -- 172.21.15.98:0/4189277677 shutdown_connections 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.583+0000 7f764db10700 1 -- 172.21.15.98:0/4189277677 wait complete. 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 Processor -- start 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 -- start start 2024-03-18T20:24:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76480ffc60 0x7f7648103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 0x7f7648102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648102f10 con 0x7f76481017f0 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648103080 con 0x7f76480ffc60 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f764db10700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481a0e60 con 0x7f7648100e50 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45224/0 (socket says 172.21.15.98:45224) 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 -- 172.21.15.98:0/3874833992 learned_addr learned my addr 172.21.15.98:0/3874833992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7646ffd700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 0x7f7648102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f76477fe700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76480ffc60 0x7f7648103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 -- 172.21.15.98:0/3874833992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 msgr2=0x7f7648102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 0x7f7648102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 -- 172.21.15.98:0/3874833992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76480ffc60 msgr2=0x7f7648103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76480ffc60 0x7f7648103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.584+0000 7f7647fff700 1 -- 172.21.15.98:0/3874833992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638009d00 con 0x7f76481017f0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f7647fff700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f764000b550 tx=0x7f764000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764000bea0 con 0x7f76481017f0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7640011cc0 con 0x7f76481017f0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76481a10f0 con 0x7f76481017f0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640010d40 con 0x7f76481017f0 2024-03-18T20:24:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.585+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76481a15b0 con 0x7f76481017f0 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.586+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648067af0 con 0x7f76481017f0 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.587+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7640011830 con 0x7f76481017f0 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.587+0000 7f7644ff9700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 0x7f76340a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.587+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f76400cca50 con 0x7f76481017f0 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.587+0000 7f76477fe700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 0x7f76340a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.588+0000 7f76477fe700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 0x7f76340a35e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7638007a30 tx=0x7f763800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:44.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.591+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7640096010 con 0x7f76481017f0 2024-03-18T20:24:44.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:44 smithi136 ceph-mon[48073]: pgmap v221: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 7 op/s 2024-03-18T20:24:44.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:44 smithi136 ceph-mon[48073]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T20:24:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:44.789+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f76481a17d0 con 0x7f76481017f0 2024-03-18T20:24:44.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:44 smithi098 ceph-mon[43462]: pgmap v221: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 7 op/s 2024-03-18T20:24:44.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:44 smithi098 ceph-mon[43462]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T20:24:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:44 smithi169 ceph-mon[48529]: pgmap v221: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 7 op/s 2024-03-18T20:24:44.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:44 smithi169 ceph-mon[48529]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T20:24:45.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.427+0000 7f7644ff9700 1 -- 172.21.15.98:0/3874833992 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7640099630 con 0x7f76481017f0 2024-03-18T20:24:45.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.441+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 msgr2=0x7f76340a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.441+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 0x7f76340a35e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7638007a30 tx=0x7f763800b040 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.441+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 msgr2=0x7f7648102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.441+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f764000b550 tx=0x7f764000b860 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 shutdown_connections 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76481017f0 0x7f7648102770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76340a1120 0x7f76340a35e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76480ffc60 0x7f7648103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 --2- 172.21.15.98:0/3874833992 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7648100e50 0x7f7648102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.442+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 >> 172.21.15.98:0/3874833992 conn(0x7f76480fb1d0 msgr2=0x7f7648067ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:45.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.444+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 shutdown_connections 2024-03-18T20:24:45.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.446+0000 7f764db10700 1 -- 172.21.15.98:0/3874833992 wait complete. 2024-03-18T20:24:45.456 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T20:24:45.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T20:24:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/2466920783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a441020b0 msgr2=0x7f5a44102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/2466920783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a441020b0 0x7f5a44102510 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009990 tx=0x7f5a38009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/2466920783 shutdown_connections 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/2466920783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a44102a50 0x7f5a44109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/2466920783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a440ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/2466920783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a441020b0 0x7f5a44102510 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.594+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/2466920783 >> 172.21.15.98:0/2466920783 conn(0x7f5a440fa950 msgr2=0x7f5a440fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/2466920783 shutdown_connections 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/2466920783 wait complete. 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 Processor -- start 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- start start 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a441019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a441020b0 0x7f5a44100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44100d40 con 0x7f5a44102a50 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44100eb0 con 0x7f5a440ff040 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.595+0000 7f5a4a6bc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a441a0c40 con 0x7f5a441020b0 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45238/0 (socket says 172.21.15.98:45238) 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 -- 172.21.15.98:0/1196423285 learned_addr learned my addr 172.21.15.98:0/1196423285 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a43fff700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a441019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a437fe700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a441020b0 0x7f5a44100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 -- 172.21.15.98:0/1196423285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a441020b0 msgr2=0x7f5a44100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a441020b0 0x7f5a44100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 -- 172.21.15.98:0/1196423285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 msgr2=0x7f5a441019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a441019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 -- 172.21.15.98:0/1196423285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380095f0 con 0x7f5a44102a50 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a43fff700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a441019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a48c59700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c007ee0 tx=0x7f5a3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c017070 con 0x7f5a44102a50 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5a3c00f040 con 0x7f5a44102a50 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a441a0ed0 con 0x7f5a44102a50 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c005570 con 0x7f5a44102a50 2024-03-18T20:24:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.596+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441a1300 con 0x7f5a44102a50 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.597+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a44108380 con 0x7f5a44102a50 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.598+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f5a3c00db70 con 0x7f5a44102a50 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.599+0000 7f5a417fa700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 0x7f5a2c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.599+0000 7f5a43fff700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 0x7f5a2c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.599+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f5a3c0cc2d0 con 0x7f5a44102a50 2024-03-18T20:24:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.599+0000 7f5a43fff700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 0x7f5a2c0a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a44067b30 tx=0x7f5a34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.603+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5a3c095830 con 0x7f5a44102a50 2024-03-18T20:24:45.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:45 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T20:24:45.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:45 smithi136 ceph-mon[48073]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T20:24:45.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:45 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T20:24:45.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:45 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T20:24:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:45 smithi098 ceph-mon[43462]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T20:24:45.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:45 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T20:24:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.849+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f5a44067260 con 0x7f5a44102a50 2024-03-18T20:24:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.857+0000 7f5a417fa700 1 -- 172.21.15.98:0/1196423285 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f5a3c098e50 con 0x7f5a44102a50 2024-03-18T20:24:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 msgr2=0x7f5a2c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 0x7f5a2c0a35e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a44067b30 tx=0x7f5a34005f90 comp rx=0 tx=0).stop 2024-03-18T20:24:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 msgr2=0x7f5a441005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c007ee0 tx=0x7f5a3c00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 shutdown_connections 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a44102a50 0x7f5a441005a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5a2c0a1120 0x7f5a2c0a35e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a440ff040 0x7f5a441019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 --2- 172.21.15.98:0/1196423285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a441020b0 0x7f5a44100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 >> 172.21.15.98:0/1196423285 conn(0x7f5a440fa950 msgr2=0x7f5a441054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 shutdown_connections 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.859+0000 7f5a4a6bc700 1 -- 172.21.15.98:0/1196423285 wait complete. 2024-03-18T20:24:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:added data pool 4 to fsmap 2024-03-18T20:24:45.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:45.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:45 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3134423370' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T20:24:45.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:45 smithi169 ceph-mon[48529]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T20:24:45.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:45 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T20:24:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 -- 172.21.15.98:0/3468193057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0101700 msgr2=0x7fd4b0107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 --2- 172.21.15.98:0/3468193057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0101700 0x7fd4b0107fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800b0d0 tx=0x7fd4a800b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 -- 172.21.15.98:0/3468193057 shutdown_connections 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 --2- 172.21.15.98:0/3468193057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0101700 0x7fd4b0107fb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 --2- 172.21.15.98:0/3468193057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b00ffb70 0x7fd4b00fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 --2- 172.21.15.98:0/3468193057 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b0100d60 0x7fd4b01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 -- 172.21.15.98:0/3468193057 >> 172.21.15.98:0/3468193057 conn(0x7fd4b00fb100 msgr2=0x7fd4b00fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 -- 172.21.15.98:0/3468193057 shutdown_connections 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.997+0000 7fd4b6209700 1 -- 172.21.15.98:0/3468193057 wait complete. 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 Processor -- start 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 -- start start 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b00ffb70 0x7fd4b0196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b0101700 0x7fd4b019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b019e940 con 0x7fd4b0100d60 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b019ea80 con 0x7fd4b0101700 2024-03-18T20:24:46.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4b6209700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b019ec10 con 0x7fd4b00ffb70 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4aeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4aeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45260/0 (socket says 172.21.15.98:45260) 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4aeffd700 1 -- 172.21.15.98:0/1483025866 learned_addr learned my addr 172.21.15.98:0/1483025866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.998+0000 7fd4af7fe700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b00ffb70 0x7fd4b0196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4affff700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b0101700 0x7fd4b019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 -- 172.21.15.98:0/1483025866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b00ffb70 msgr2=0x7fd4b0196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b00ffb70 0x7fd4b0196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 -- 172.21.15.98:0/1483025866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b0101700 msgr2=0x7fd4b019e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b0101700 0x7fd4b019e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 -- 172.21.15.98:0/1483025866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8009d00 con 0x7fd4b0100d60 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4aeffd700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b530 tx=0x7fd4a400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4011830 con 0x7fd4b0100d60 2024-03-18T20:24:46.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4a4011e20 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b019eef0 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a400f2e0 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:45.999+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b019f440 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.000+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b0067e60 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.001+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd4a4011990 con 0x7fd4b0100d60 2024-03-18T20:24:46.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.002+0000 7fd4acff9700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 0x7fd4980a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.002+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd4a40cc940 con 0x7fd4b0100d60 2024-03-18T20:24:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.002+0000 7fd4af7fe700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 0x7fd4980a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.002+0000 7fd4af7fe700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 0x7fd4980a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0100bc0 tx=0x7fd4a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.005+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4a4095ee0 con 0x7fd4b0100d60 2024-03-18T20:24:46.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.246+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4b019f800 con 0x7fd4b0100d60 2024-03-18T20:24:46.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.246+0000 7fd4acff9700 1 -- 172.21.15.98:0/1483025866 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10816 (secure 0 0 0) 0x7fd4a4099500 con 0x7fd4b0100d60 2024-03-18T20:24:46.250 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:46.250 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:24:45.850273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-18T20:24:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 msgr2=0x7fd4980a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 0x7fd4980a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0100bc0 tx=0x7fd4a0009190 comp rx=0 tx=0).stop 2024-03-18T20:24:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 msgr2=0x7fd4b01973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b530 tx=0x7fd4a400b840 comp rx=0 tx=0).stop 2024-03-18T20:24:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 shutdown_connections 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4980a11a0 0x7fd4980a3660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b0101700 0x7fd4b019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4b00ffb70 0x7fd4b0196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 --2- 172.21.15.98:0/1483025866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4b0100d60 0x7fd4b01973a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.248+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 >> 172.21.15.98:0/1483025866 conn(0x7fd4b00fb100 msgr2=0x7fd4b0069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.249+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 shutdown_connections 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.249+0000 7fd4b6209700 1 -- 172.21.15.98:0/1483025866 wait complete. 2024-03-18T20:24:46.253 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2024-03-18T20:24:46.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 -- 172.21.15.98:0/4103279405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4102820 msgr2=0x7facb41090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 --2- 172.21.15.98:0/4103279405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4102820 0x7facb41090d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7facac00b0d0 tx=0x7facac00b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 -- 172.21.15.98:0/4103279405 shutdown_connections 2024-03-18T20:24:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 --2- 172.21.15.98:0/4103279405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4102820 0x7facb41090d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 --2- 172.21.15.98:0/4103279405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4100c90 0x7facb41010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 --2- 172.21.15.98:0/4103279405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4101e80 0x7facb41022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 -- 172.21.15.98:0/4103279405 >> 172.21.15.98:0/4103279405 conn(0x7facb40fc200 msgr2=0x7facb40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 -- 172.21.15.98:0/4103279405 shutdown_connections 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.382+0000 7facb8ecb700 1 -- 172.21.15.98:0/4103279405 wait complete. 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 Processor -- start 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 -- start start 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4101e80 0x7facb4197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4102820 0x7facb419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb419e730 con 0x7facb4100c90 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb419e870 con 0x7facb4101e80 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb8ecb700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb419ea00 con 0x7facb4102820 2024-03-18T20:24:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45278/0 (socket says 172.21.15.98:45278) 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 -- 172.21.15.98:0/1774737745 learned_addr learned my addr 172.21.15.98:0/1774737745 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb1d9b700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4101e80 0x7facb4197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb2d9d700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4102820 0x7facb419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 -- 172.21.15.98:0/1774737745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4102820 msgr2=0x7facb419e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4102820 0x7facb419e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 -- 172.21.15.98:0/1774737745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4101e80 msgr2=0x7facb4197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4101e80 0x7facb4197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 -- 172.21.15.98:0/1774737745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009d00 con 0x7facb4100c90 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.383+0000 7facb259c700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faca400d980 tx=0x7faca400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca4004020 con 0x7facb4100c90 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faca400f690 con 0x7facb4100c90 2024-03-18T20:24:46.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb419ece0 con 0x7facb4100c90 2024-03-18T20:24:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca40039e0 con 0x7facb4100c90 2024-03-18T20:24:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb419f1a0 con 0x7facb4100c90 2024-03-18T20:24:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.384+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb4108160 con 0x7facb4100c90 2024-03-18T20:24:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.385+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7faca400f800 con 0x7facb4100c90 2024-03-18T20:24:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.386+0000 7facab7fe700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 0x7faca00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.386+0000 7facb1d9b700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 0x7faca00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.386+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faca40cc9e0 con 0x7facb4100c90 2024-03-18T20:24:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.386+0000 7facb1d9b700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 0x7faca00a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009c60 tx=0x7fac9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.390+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faca4095f80 con 0x7facb4100c90 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: pgmap v224: 129 pgs: 18 creating+peering, 21 unknown, 90 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:46.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:46 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1483025866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:46.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.599+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb402ce40 con 0x7facb4100c90 2024-03-18T20:24:46.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.601+0000 7facab7fe700 1 -- 172.21.15.98:0/1774737745 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7faca40995a0 con 0x7facb4100c90 2024-03-18T20:24:46.605 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:46.605 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:46.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 msgr2=0x7faca00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 0x7faca00a35e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009c60 tx=0x7fac9c009260 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 msgr2=0x7facb4196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faca400d980 tx=0x7faca400dc90 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 shutdown_connections 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7facb4102820 0x7facb419e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facb4100c90 0x7facb4196c00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faca00a1120 0x7faca00a35e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 --2- 172.21.15.98:0/1774737745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb4101e80 0x7facb4197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 >> 172.21.15.98:0/1774737745 conn(0x7facb40fc200 msgr2=0x7facb41052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 shutdown_connections 2024-03-18T20:24:46.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.603+0000 7facb8ecb700 1 -- 172.21.15.98:0/1774737745 wait complete. 2024-03-18T20:24:46.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- 172.21.15.98:0/990093651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc0ffb60 msgr2=0x7f56bc0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 --2- 172.21.15.98:0/990093651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc0ffb60 0x7f56bc0fff80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0099c0 tx=0x7f56ac009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- 172.21.15.98:0/990093651 shutdown_connections 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 --2- 172.21.15.98:0/990093651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc1016f0 0x7f56bc107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 --2- 172.21.15.98:0/990093651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc0ffb60 0x7f56bc0fff80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 --2- 172.21.15.98:0/990093651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc100d50 0x7f56bc1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- 172.21.15.98:0/990093651 >> 172.21.15.98:0/990093651 conn(0x7f56bc0fb0d0 msgr2=0x7f56bc0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- 172.21.15.98:0/990093651 shutdown_connections 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- 172.21.15.98:0/990093651 wait complete. 2024-03-18T20:24:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 Processor -- start 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.737+0000 7f56c3916700 1 -- start start 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc0ffb60 0x7f56bc195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 0x7f56bc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc19d550 con 0x7f56bc100d50 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc19d690 con 0x7f56bc1016f0 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc19d820 con 0x7f56bc0ffb60 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45298/0 (socket says 172.21.15.98:45298) 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 -- 172.21.15.98:0/2203307479 learned_addr learned my addr 172.21.15.98:0/2203307479 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c16b2700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc0ffb60 0x7f56bc195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c1eb3700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 0x7f56bc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 -- 172.21.15.98:0/2203307479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc0ffb60 msgr2=0x7f56bc195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc0ffb60 0x7f56bc195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 -- 172.21.15.98:0/2203307479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 msgr2=0x7f56bc19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 0x7f56bc19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 -- 172.21.15.98:0/2203307479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac0095f0 con 0x7f56bc100d50 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c1eb3700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 0x7f56bc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c0eb1700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56b000c920 tx=0x7f56b000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0007b00 con 0x7f56bc100d50 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc19db00 con 0x7f56bc100d50 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f56b0004cc0 con 0x7f56bc100d50 2024-03-18T20:24:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0005490 con 0x7f56bc100d50 2024-03-18T20:24:46.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.738+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc19e010 con 0x7f56bc100d50 2024-03-18T20:24:46.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.739+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc107080 con 0x7f56bc100d50 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: pgmap v224: 129 pgs: 18 creating+peering, 21 unknown, 90 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T20:24:46.747 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T20:24:46.748 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:46.748 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:46 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1483025866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.740+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f56b0004750 con 0x7f56bc100d50 2024-03-18T20:24:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.741+0000 7f56ba7fc700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 0x7f56a40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.741+0000 7f56c16b2700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 0x7f56a40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.741+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f56b00cc6e0 con 0x7f56bc100d50 2024-03-18T20:24:46.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.741+0000 7f56c16b2700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 0x7f56a40a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56ac000c00 tx=0x7f56ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.745+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f56b0095c80 con 0x7f56bc100d50 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3874833992' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: pgmap v224: 129 pgs: 18 creating+peering, 21 unknown, 90 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1196423285' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:46.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:46 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1483025866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:46.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.949+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56bc067a10 con 0x7f56bc100d50 2024-03-18T20:24:46.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.951+0000 7f56ba7fc700 1 -- 172.21.15.98:0/2203307479 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f56b00992a0 con 0x7f56bc100d50 2024-03-18T20:24:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:46.955 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 msgr2=0x7f56a40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 0x7f56a40a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56ac000c00 tx=0x7f56ac005fd0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 msgr2=0x7f56bc195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56b000c920 tx=0x7f56b000cce0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 shutdown_connections 2024-03-18T20:24:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f56a40a1120 0x7f56a40a35e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56bc1016f0 0x7f56bc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56bc0ffb60 0x7f56bc195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 --2- 172.21.15.98:0/2203307479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56bc100d50 0x7f56bc195f60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 >> 172.21.15.98:0/2203307479 conn(0x7f56bc0fb0d0 msgr2=0x7f56bc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 shutdown_connections 2024-03-18T20:24:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:46.953+0000 7f56c3916700 1 -- 172.21.15.98:0/2203307479 wait complete. 2024-03-18T20:24:46.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:24:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/4154562140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 msgr2=0x7fa3b8103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/4154562140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8103ad0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009990 tx=0x7fa3b4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/4154562140 shutdown_connections 2024-03-18T20:24:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/4154562140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8106940 0x7fa3b8108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/4154562140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8103ad0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/4154562140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8104010 0x7fa3b8106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/4154562140 >> 172.21.15.98:0/4154562140 conn(0x7fa3b80fb0d0 msgr2=0x7fa3b80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/4154562140 shutdown_connections 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.101+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/4154562140 wait complete. 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 Processor -- start 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 -- start start 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 0x7fa3b81971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 0x7fa3b819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b819e760 con 0x7fa3b81016e0 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b819e8d0 con 0x7fa3b8106940 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bfc3b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b819ea90 con 0x7fa3b8104010 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45314/0 (socket says 172.21.15.98:45314) 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 -- 172.21.15.98:0/2806166016 learned_addr learned my addr 172.21.15.98:0/2806166016 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3be1d8700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 0x7fa3b819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 -- 172.21.15.98:0/2806166016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 msgr2=0x7fa3b81971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd1d6700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 0x7fa3b81971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 0x7fa3b81971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 -- 172.21.15.98:0/2806166016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 msgr2=0x7fa3b819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 0x7fa3b819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd9d7700 1 -- 172.21.15.98:0/2806166016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b40095f0 con 0x7fa3b81016e0 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3be1d8700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 0x7fa3b819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.102+0000 7fa3bd1d6700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 0x7fa3b81971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3bd9d7700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b40038d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b401d070 con 0x7fa3b81016e0 2024-03-18T20:24:47.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa3b4022470 con 0x7fa3b81016e0 2024-03-18T20:24:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b819ed10 con 0x7fa3b81016e0 2024-03-18T20:24:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b400f6c0 con 0x7fa3b81016e0 2024-03-18T20:24:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.103+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b819f130 con 0x7fa3b81016e0 2024-03-18T20:24:47.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.104+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b80fcc50 con 0x7fa3b81016e0 2024-03-18T20:24:47.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.105+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa3b40225e0 con 0x7fa3b81016e0 2024-03-18T20:24:47.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.105+0000 7fa3aeffd700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 0x7fa3a40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.105+0000 7fa3bd1d6700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 0x7fa3a40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.105+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa3b40ce4f0 con 0x7fa3b81016e0 2024-03-18T20:24:47.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.106+0000 7fa3bd1d6700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 0x7fa3a40a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8007900 tx=0x7fa3a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.109+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa3b4097b20 con 0x7fa3b81016e0 2024-03-18T20:24:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.349+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3b8067a10 con 0x7fa3b81016e0 2024-03-18T20:24:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.350+0000 7fa3aeffd700 1 -- 172.21.15.98:0/2806166016 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10816 (secure 0 0 0) 0x7fa3b409b140 con 0x7fa3b81016e0 2024-03-18T20:24:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:47.354 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:24:45.850273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 msgr2=0x7fa3a40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 0x7fa3a40a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8007900 tx=0x7fa3a8008040 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 msgr2=0x7fa3b8196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b40038d0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 shutdown_connections 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa3a40a1120 0x7fa3a40a35e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3b8106940 0x7fa3b819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b81016e0 0x7fa3b8196c70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 --2- 172.21.15.98:0/2806166016 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b8104010 0x7fa3b81971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 >> 172.21.15.98:0/2806166016 conn(0x7fa3b80fb0d0 msgr2=0x7fa3b8102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 shutdown_connections 2024-03-18T20:24:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.352+0000 7fa3bfc3b700 1 -- 172.21.15.98:0/2806166016 wait complete. 2024-03-18T20:24:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2024-03-18T20:24:47.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/3664722688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f41016f0 msgr2=0x7fe0f4107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/3664722688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f41016f0 0x7fe0f4107fa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b0d0 tx=0x7fe0ec00b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/3664722688 shutdown_connections 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/3664722688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f41016f0 0x7fe0f4107fa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/3664722688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 0x7fe0f40fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/3664722688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f4100d50 0x7fe0f41011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/3664722688 >> 172.21.15.98:0/3664722688 conn(0x7fe0f40fb0d0 msgr2=0x7fe0f40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/3664722688 shutdown_connections 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/3664722688 wait complete. 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 Processor -- start 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.487+0000 7fe0fb6c4700 1 -- start start 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 0x7fe0f4196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f41016f0 0x7fe0f419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419e7e0 con 0x7fe0f4100d50 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419e950 con 0x7fe0f40ffb60 2024-03-18T20:24:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419eb10 con 0x7fe0f41016f0 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45332/0 (socket says 172.21.15.98:45332) 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 -- 172.21.15.98:0/236286957 learned_addr learned my addr 172.21.15.98:0/236286957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f9c61700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f41016f0 0x7fe0f419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f9460700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 0x7fe0f4196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 -- 172.21.15.98:0/236286957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f41016f0 msgr2=0x7fe0f419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f41016f0 0x7fe0f419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 -- 172.21.15.98:0/236286957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 msgr2=0x7fe0f4196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 0x7fe0f4196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 -- 172.21.15.98:0/236286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec009d00 con 0x7fe0f4100d50 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0f8c5f700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800b530 tx=0x7fe0e800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8011830 con 0x7fe0f4100d50 2024-03-18T20:24:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe0e8011e20 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f419edf0 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e800f2e0 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.488+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f419f300 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.489+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f4067a10 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.490+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe0e8011990 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.491+0000 7fe0e67fc700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 0x7fe0e00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.491+0000 7fe0f9460700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 0x7fe0e00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.491+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe0e80cc9c0 con 0x7fe0f4100d50 2024-03-18T20:24:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.491+0000 7fe0f9460700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 0x7fe0e00a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005fd0 tx=0x7fe0f0005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.495+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0e8095f40 con 0x7fe0f4100d50 2024-03-18T20:24:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.702+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0f419f580 con 0x7fe0f4100d50 2024-03-18T20:24:47.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.704+0000 7fe0e67fc700 1 -- 172.21.15.98:0/236286957 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7fe0e8015090 con 0x7fe0f4100d50 2024-03-18T20:24:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:47.708 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:47.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 msgr2=0x7fe0e00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 0x7fe0e00a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005fd0 tx=0x7fe0f0005dd0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 msgr2=0x7fe0f4197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800b530 tx=0x7fe0e800b840 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 shutdown_connections 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe0f41016f0 0x7fe0f419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe0e00a11a0 0x7fe0e00a3660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0f40ffb60 0x7fe0f4196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 --2- 172.21.15.98:0/236286957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0f4100d50 0x7fe0f4197200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 >> 172.21.15.98:0/236286957 conn(0x7fe0f40fb0d0 msgr2=0x7fe0f40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 shutdown_connections 2024-03-18T20:24:47.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.706+0000 7fe0fb6c4700 1 -- 172.21.15.98:0/236286957 wait complete. 2024-03-18T20:24:47.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:47 smithi098 ceph-mon[43462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T20:24:47.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:47 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1774737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.721 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:47 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2203307479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.721 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:47 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2806166016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:24:41.335739+0000', 'modified': '2024-03-18T20:24:45.850273+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.136:6839/2871006308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 2871006308}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 2871006308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-18T20:24:47.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:24:47.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T20:24:47.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:47 smithi136 ceph-mon[48073]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T20:24:47.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:47 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1774737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:47 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2203307479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:47 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2806166016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 -- 172.21.15.98:0/349264096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 msgr2=0x7f12941011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 --2- 172.21.15.98:0/349264096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f12941011a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f12880099c0 tx=0x7f1288009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 -- 172.21.15.98:0/349264096 shutdown_connections 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 --2- 172.21.15.98:0/349264096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294102910 0x7f12941091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 --2- 172.21.15.98:0/349264096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f12941011a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 --2- 172.21.15.98:0/349264096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294101f70 0x7f12941023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.841+0000 7f129a7c1700 1 -- 172.21.15.98:0/349264096 >> 172.21.15.98:0/349264096 conn(0x7f12940fc2f0 msgr2=0x7f12940fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- 172.21.15.98:0/349264096 shutdown_connections 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- 172.21.15.98:0/349264096 wait complete. 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 Processor -- start 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- start start 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294101f70 0x7f12940731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294102910 0x7f12940736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294073e80 con 0x7f1294100d80 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294073ff0 con 0x7f1294102910 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f129a7c1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a0c90 con 0x7f1294101f70 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.842+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45358/0 (socket says 172.21.15.98:45358) 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 -- 172.21.15.98:0/597057088 learned_addr learned my addr 172.21.15.98:0/597057088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 -- 172.21.15.98:0/597057088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294101f70 msgr2=0x7f12940731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294101f70 0x7f12940731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 -- 172.21.15.98:0/597057088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294102910 msgr2=0x7f12940736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1298d5e700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294102910 0x7f12940736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294102910 0x7f12940736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 -- 172.21.15.98:0/597057088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12880095f0 con 0x7f1294100d80 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f1293fff700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1288009f50 tx=0x7f1288005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f12917fa700 1 -- 172.21.15.98:0/597057088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128801d070 con 0x7f1294100d80 2024-03-18T20:24:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f12917fa700 1 -- 172.21.15.98:0/597057088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f128800bda0 con 0x7f1294100d80 2024-03-18T20:24:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12941a0ec0 con 0x7f1294100d80 2024-03-18T20:24:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f12917fa700 1 -- 172.21.15.98:0/597057088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288022d20 con 0x7f1294100d80 2024-03-18T20:24:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.843+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12941a1280 con 0x7f1294100d80 2024-03-18T20:24:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.845+0000 7f12917fa700 1 -- 172.21.15.98:0/597057088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f1288003f80 con 0x7f1294100d80 2024-03-18T20:24:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.846+0000 7f12917fa700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 0x7f127c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.846+0000 7f12937fe700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 0x7f127c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.846+0000 7f12917fa700 1 -- 172.21.15.98:0/597057088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f12880cde90 con 0x7f1294100d80 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.846+0000 7f12937fe700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 0x7f127c0a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1284005950 tx=0x7f12840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.846+0000 7f12a6956b80 1 Processor -- start 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 -- start start 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ac530 0x55c7934acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793404940 con 0x55c7934ad3d0 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793402300 con 0x55c7934abbd0 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12a6956b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7933fdbf0 con 0x55c7934ac530 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45360/0 (socket says 172.21.15.98:45360) 2024-03-18T20:24:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 -- 172.21.15.98:0/2730232045 learned_addr learned my addr 172.21.15.98:0/2730232045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1293fff700 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f12937fe700 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ac530 0x55c7934acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 -- 172.21.15.98:0/2730232045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ac530 msgr2=0x55c7934acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ac530 0x55c7934acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 -- 172.21.15.98:0/2730232045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 msgr2=0x55c7934abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 -- 172.21.15.98:0/2730232045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c79327f360 con 0x55c7934ad3d0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f1298d5e700 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f128c0060b0 tx=0x7f128c007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e40d44a8b602fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f127a7fc700 1 -- 172.21.15.98:0/2730232045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c003a00 con 0x55c7934ad3d0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f127a7fc700 1 -- 172.21.15.98:0/2730232045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f128c004020 con 0x55c7934ad3d0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.847+0000 7f127a7fc700 1 -- 172.21.15.98:0/2730232045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c014660 con 0x55c7934ad3d0 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- 172.21.15.98:0/2730232045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 msgr2=0x55c7934ad850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f128c0060b0 tx=0x7f128c007500 comp rx=0 tx=0).stop 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- 172.21.15.98:0/2730232045 shutdown_connections 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934ad850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 --2- 172.21.15.98:0/2730232045 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ac530 0x55c7934acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- 172.21.15.98:0/2730232045 >> 172.21.15.98:0/2730232045 conn(0x55c7934211d0 msgr2=0x55c7934a6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- 172.21.15.98:0/2730232045 shutdown_connections 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- 172.21.15.98:0/2730232045 wait complete. 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.848+0000 7f12a6956b80 1 -- start start 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 Processor -- start 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934abbd0 0x55c7934b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793399d30 con 0x55c7934ad3d0 2024-03-18T20:24:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7933999d0 con 0x55c7934ac530 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793281650 con 0x55c7934abbd0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12a6956b80 10 client.0 _ll_register_callbacks cb 0x55c793205350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45372/0 (socket says 172.21.15.98:45372) 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 -- 172.21.15.98:0/3212200732 learned_addr learned my addr 172.21.15.98:0/3212200732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 -- 172.21.15.98:0/3212200732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934abbd0 msgr2=0x55c7934b5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934abbd0 0x55c7934b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 -- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 msgr2=0x55c7934b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 -- 172.21.15.98:0/3212200732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c009920 con 0x55c7934ad3d0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f1298d5e700 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f128c006080 tx=0x7f128c004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d53ce805ce7df91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.849+0000 7f12737fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:24:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c01e460 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f128c01e5c0 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c79349ea80 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f128c01e780 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 10 client.14718 fetch_fsmap learned FSMap version 16 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55c79322bb10 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55c79349ea80 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f128c01c040 con 0x55c7934ad3d0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 10 client.14718 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 10 client.14718 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 20 client.14718 populate_metadata read hostname 'smithi098' 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c7934c0bc0 0x55c7934c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 4 client.14718 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.850+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7933f66b0 con 0x55c7934c0bc0 2024-03-18T20:24:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.851+0000 7f1293fff700 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c7934c0bc0 0x55c7934c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.851+0000 7f1293fff700 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c7934c0bc0 0x55c7934c5f20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ff2395b2f98bdf server_cookie=feeaa406c1c6d6a4 in_seq=0 out_seq=0 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.851+0000 7f12737fe700 10 client.14718 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12737fe700 1 -- 172.21.15.98:0/3212200732 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c7933f66b0 con 0x55c7934c0bc0 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12737fe700 10 client.14718 handle_command_reply: tid=0 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 client.14718 shutdown 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 msgr2=0x55c7934bd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f128c006080 tx=0x7f128c004550 comp rx=0 tx=0).stop 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 shutdown_connections 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c7934c0bc0 0x55c7934c5f20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ad3d0 0x55c7934bd020 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934abbd0 0x55c7934b5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12737fe700 0 client.14718 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3212200732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 >> 172.21.15.98:0/3212200732 conn(0x55c7934211d0 msgr2=0x55c7934a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 shutdown_connections 2024-03-18T20:24:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 1 -- 172.21.15.98:0/3212200732 wait complete. 2024-03-18T20:24:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.852+0000 7f12a6956b80 20 client.14718 trim_cache size 0 max 16384 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 Processor -- start 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 -- start start 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934bcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7934ad280 con 0x55c7934abbd0 2024-03-18T20:24:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7933f66e0 con 0x55c7934ac530 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12a6956b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7933fa850 con 0x55c7934ad3d0 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45384/0 (socket says 172.21.15.98:45384) 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 -- 172.21.15.98:0/3750200170 learned_addr learned my addr 172.21.15.98:0/3750200170 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f12937fe700 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 -- 172.21.15.98:0/3750200170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 msgr2=0x55c7934bcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934bcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 -- 172.21.15.98:0/3750200170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 msgr2=0x55c7934b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 -- 172.21.15.98:0/3750200170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7933c3e50 con 0x55c7934abbd0 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f1293fff700 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f12880cbb10 tx=0x7f12880cb670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08426c5660dabf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f127affd700 1 -- 172.21.15.98:0/3750200170 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128809efb0 con 0x55c7934abbd0 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.870+0000 7f127affd700 1 -- 172.21.15.98:0/3750200170 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12880a3570 con 0x55c7934abbd0 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f127affd700 1 -- 172.21.15.98:0/3750200170 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128809b640 con 0x55c7934abbd0 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- 172.21.15.98:0/3750200170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 msgr2=0x55c7934b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f12880cbb10 tx=0x7f12880cb670 comp rx=0 tx=0).stop 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- 172.21.15.98:0/3750200170 shutdown_connections 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934bcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934abbd0 0x55c7934b02d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 --2- 172.21.15.98:0/3750200170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934ac530 0x55c7934b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- 172.21.15.98:0/3750200170 >> 172.21.15.98:0/3750200170 conn(0x55c7934211d0 msgr2=0x55c7934a6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- 172.21.15.98:0/3750200170 shutdown_connections 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- 172.21.15.98:0/3750200170 wait complete. 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.871+0000 7f12a6956b80 1 -- start start 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 Processor -- start 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7934b8530 con 0x55c7934ac530 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793515510 con 0x55c7934abbd0 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c793515650 con 0x55c7934ad3d0 2024-03-18T20:24:47.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12a6956b80 10 client.0 _ll_register_callbacks cb 0x7f1288004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45398/0 (socket says 172.21.15.98:45398) 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 -- 172.21.15.98:0/413745286 learned_addr learned my addr 172.21.15.98:0/413745286 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f1293fff700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f1298d5e700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 -- 172.21.15.98:0/413745286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 msgr2=0x55c7934b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934b7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 -- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 msgr2=0x55c7934b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f12937fe700 1 -- 172.21.15.98:0/413745286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12941a1280 con 0x55c7934ac530 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.872+0000 7f1293fff700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12937fe700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f128400de70 tx=0x7f1284010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a8a6938ffdf901 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284010e80 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f128401e480 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f128c004020 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f128401e640 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 10 client.14724 fetch_fsmap learned FSMap version 16 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55c793283c80 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1284032080 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f128402e760 con 0x55c7934ac530 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 10 client.14724 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 10 client.14724 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 20 client.14724 populate_metadata read hostname 'smithi098' 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c79351a510 0x55c793516ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 4 client.14724 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.873+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c79324cca0 con 0x55c79351a510 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f1293fff700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c79351a510 0x55c793516ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f12737fe700 10 client.14724 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f1293fff700 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c79351a510 0x55c793516ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d264c33b610b4a00 server_cookie=f0aabb4def36d78c in_seq=0 out_seq=0 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f12737fe700 1 -- 172.21.15.98:0/413745286 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c79324cca0 con 0x55c79351a510 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f12737fe700 10 client.14724 handle_command_reply: tid=0 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.874+0000 7f12a6956b80 1 client.14724 shutdown 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 msgr2=0x55c7934bcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f128400de70 tx=0x7f1284010740 comp rx=0 tx=0).stop 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 shutdown_connections 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x55c79351a510 0x55c793516ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c7934ad3d0 0x55c7934b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12737fe700 0 client.14724 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7934abbd0 0x55c7934b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 --2- 172.21.15.98:0/413745286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7934ac530 0x55c7934bcf10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 >> 172.21.15.98:0/413745286 conn(0x55c7934211d0 msgr2=0x55c7934b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 shutdown_connections 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 1 -- 172.21.15.98:0/413745286 wait complete. 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f12a6956b80 20 client.14724 trim_cache size 0 max 16384 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-18T20:24:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24494, 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 6.5365257799999998, 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 84, 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 12.189636372000001 2024-03-18T20:24:47.887 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 msgr2=0x7f127c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 0x7f127c0a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1284005950 tx=0x7f12840058e0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 msgr2=0x7f1294074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1288009f50 tx=0x7f1288005b00 comp rx=0 tx=0).stop 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 shutdown_connections 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f127c0a1120 0x7f127c0a35e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1294102910 0x7f12940736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1294100d80 0x7f1294074b10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 --2- 172.21.15.98:0/597057088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294101f70 0x7f12940731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 >> 172.21.15.98:0/597057088 conn(0x7f12940fc2f0 msgr2=0x7f1294105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 shutdown_connections 2024-03-18T20:24:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:47.875+0000 7f129a7c1700 1 -- 172.21.15.98:0/597057088 wait complete. 2024-03-18T20:24:47.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:47 smithi169 ceph-mon[48529]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T20:24:47.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:47 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1774737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:47 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2203307479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.889 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:47 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2806166016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:24:47.891 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T20:24:47.891 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:47.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T20:24:48.110 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:24:48.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:48 smithi098 ceph-mon[43462]: pgmap v226: 129 pgs: 15 creating+peering, 13 unknown, 101 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-18T20:24:48.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:48 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/236286957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:48.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:48 smithi136 ceph-mon[48073]: pgmap v226: 129 pgs: 15 creating+peering, 13 unknown, 101 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-18T20:24:48.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:48 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/236286957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:48.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:48 smithi169 ceph-mon[48529]: pgmap v226: 129 pgs: 15 creating+peering, 13 unknown, 101 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 4 op/s 2024-03-18T20:24:48.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:48 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/236286957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 -- 172.21.15.98:0/2757730555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b0106e10 msgr2=0x7f48b01071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 --2- 172.21.15.98:0/2757730555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b0106e10 0x7f48b01071f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009990 tx=0x7f48a4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 -- 172.21.15.98:0/2757730555 shutdown_connections 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 --2- 172.21.15.98:0/2757730555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b00690f0 0x7f48b010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 --2- 172.21.15.98:0/2757730555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b0106e10 0x7f48b01071f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 --2- 172.21.15.98:0/2757730555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b0068750 0x7f48b0068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 -- 172.21.15.98:0/2757730555 >> 172.21.15.98:0/2757730555 conn(0x7f48b0075d50 msgr2=0x7f48b0076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 -- 172.21.15.98:0/2757730555 shutdown_connections 2024-03-18T20:24:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.061+0000 7f48b721f700 1 -- 172.21.15.98:0/2757730555 wait complete. 2024-03-18T20:24:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 Processor -- start 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 -- start start 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b0068750 0x7f48b01971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b0106e10 0x7f48b019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c0f0 con 0x7f48b00690f0 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c260 con 0x7f48b0106e10 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48b721f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c420 con 0x7f48b0068750 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45402/0 (socket says 172.21.15.98:45402) 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.062+0000 7f48affff700 1 -- 172.21.15.98:0/2108420020 learned_addr learned my addr 172.21.15.98:0/2108420020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 -- 172.21.15.98:0/2108420020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b0068750 msgr2=0x7f48b01971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b0068750 0x7f48b01971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 -- 172.21.15.98:0/2108420020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b0106e10 msgr2=0x7f48b019bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b0106e10 0x7f48b019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 -- 172.21.15.98:0/2108420020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48affff700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b530 tx=0x7f489c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c011830 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b019c700 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f489c011e20 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c010dd0 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.063+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b01a3f60 con 0x7f48b00690f0 2024-03-18T20:24:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.064+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0109330 con 0x7f48b00690f0 2024-03-18T20:24:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.065+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f489c011990 con 0x7f48b00690f0 2024-03-18T20:24:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.065+0000 7f48adffb700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.066+0000 7f48b4fbb700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.066+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f489c0cc6b0 con 0x7f48b00690f0 2024-03-18T20:24:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.066+0000 7f48b4fbb700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f48a40098e0 tx=0x7f48a4003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.070+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f489c095bf0 con 0x7f48b00690f0 2024-03-18T20:24:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.305+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f48b0198500 con 0x7f48b00690f0 2024-03-18T20:24:49.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.305+0000 7f48adffb700 1 -- 172.21.15.98:0/2108420020 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f489c099210 con 0x7f48b00690f0 2024-03-18T20:24:49.309 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 msgr2=0x7f48a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f48a40098e0 tx=0x7f48a4003680 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 msgr2=0x7f48b0197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b530 tx=0x7f489c00b8f0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 shutdown_connections 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b0106e10 0x7f48b019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b0068750 0x7f48b01971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 --2- 172.21.15.98:0/2108420020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b00690f0 0x7f48b0197730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:49.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 >> 172.21.15.98:0/2108420020 conn(0x7f48b0075d50 msgr2=0x7f48b00fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 shutdown_connections 2024-03-18T20:24:49.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:49.307+0000 7f48b721f700 1 -- 172.21.15.98:0/2108420020 wait complete. 2024-03-18T20:24:49.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:49 smithi098 ceph-mon[43462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T20:24:49.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:49 smithi098 ceph-mon[43462]: Cluster is now healthy 2024-03-18T20:24:49.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:49 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2108420020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:24:49.693 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:24:49.694 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T20:24:49.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.694 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-03-18T20:24:49.712 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:24:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:24:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:24:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:24:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T20:24:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41934sec preferred_lft 41934sec 2024-03-18T20:24:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::6960:dbbe:1345:fedd/64 scope link noprefixroute 2024-03-18T20:24:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:24:49.714 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T20:24:49.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T20:24:49.715 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:24:49.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:49 smithi136 ceph-mon[48073]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T20:24:49.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:49 smithi136 ceph-mon[48073]: Cluster is now healthy 2024-03-18T20:24:49.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:49 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2108420020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:24:49.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.881 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:24:49.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:49 smithi169 ceph-mon[48529]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T20:24:49.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:49 smithi169 ceph-mon[48529]: Cluster is now healthy 2024-03-18T20:24:49.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:49 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2108420020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:24:49.910 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-18T20:24:49.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.912 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-03-18T20:24:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-03-18T20:24:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:24:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T20:24:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T20:24:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:24:49.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T20:24:49.986 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:24:50.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.216 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-18T20:24:50.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.234 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-03-18T20:24:50.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.291 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:24:50.292 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:24:50.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T20:24:50.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T20:24:50.292 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:24:50.387 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T20:24:50.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.387 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:24:50.387 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:24:50.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T20:24:50.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T20:24:50.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T20:24:50.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T20:24:50.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T20:24:50.388 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:24:50.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:50 smithi098 ceph-mon[43462]: pgmap v227: 129 pgs: 15 creating+peering, 114 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-18T20:24:50.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.584 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:24:50.584 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:24:50.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-03-18T20:24:50.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T20:24:50.584 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:24:50.682 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T20:24:50.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:24:50.683 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:50.700 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T20:24:50.700 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:24:50.700 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:50.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:50 smithi136 ceph-mon[48073]: pgmap v227: 129 pgs: 15 creating+peering, 114 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-18T20:24:50.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-18T20:24:50.825 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:24:50.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:24:50.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:50 smithi169 ceph-mon[48529]: pgmap v227: 129 pgs: 15 creating+peering, 114 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-18T20:24:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:24:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:24:50.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.893 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-18T20:24:50.950 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:24:50.950 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T20:24:50.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-18T20:24:51.019 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:24:51.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:24:51.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:2024-03-18T20:24:51.077+0000 7f1c080965c0 -1 init, newargv = 0x55ff25eceb60 newargc=15 2024-03-18T20:24:51.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[97242]: starting ceph client 2024-03-18T20:24:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:24:51.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[97242]: starting fuse 2024-03-18T20:24:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:24:51.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.088 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-18T20:24:51.145 INFO:teuthology.orchestra.run.smithi098.stdout:124 2024-03-18T20:24:51.145 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-03-18T20:24:51.146 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.admin")) 2024-03-18T20:24:51.147 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-03-18T20:24:51.269 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.admin.97242.asok 2024-03-18T20:24:51.451 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.97242.asok 2024-03-18T20:24:51.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.97242.asok status 2024-03-18T20:24:51.610 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:24:51.610 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "admin", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "97242", 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14736, 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-03-18T20:24:51.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 14736 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 524848908 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 524848908 2024-03-18T20:24:51.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.14736 192.168.0.1:0/524848908", 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/524848908", 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 16, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 85, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-03-18T20:24:51.613 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:24:51.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.621 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-03-18T20:24:51.621 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:51.638 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-03-18T20:24:51.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:51.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:51.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.707 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T20:24:51.810 INFO:teuthology.orchestra.run.smithi098.stdout:# file: . 2024-03-18T20:24:51.810 INFO:teuthology.orchestra.run.smithi098.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T20:24:51.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:51.810 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:24:51.810 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:24:51.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 -- 172.21.15.98:0/705975205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc100c90 msgr2=0x7f97fc1010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 --2- 172.21.15.98:0/705975205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc100c90 0x7f97fc1010b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97f80099c0 tx=0x7f97f8009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 -- 172.21.15.98:0/705975205 shutdown_connections 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 --2- 172.21.15.98:0/705975205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc1090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 --2- 172.21.15.98:0/705975205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc100c90 0x7f97fc1010b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 --2- 172.21.15.98:0/705975205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc101e80 0x7f97fc1022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 -- 172.21.15.98:0/705975205 >> 172.21.15.98:0/705975205 conn(0x7f97fc0fc200 msgr2=0x7f97fc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 -- 172.21.15.98:0/705975205 shutdown_connections 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.988+0000 7f980413e700 1 -- 172.21.15.98:0/705975205 wait complete. 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 Processor -- start 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 -- start start 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc100c90 0x7f97fc074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc073de0 con 0x7f97fc101e80 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc073f50 con 0x7f97fc100c90 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f980413e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1a0c60 con 0x7f97fc102820 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45414/0 (socket says 172.21.15.98:45414) 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 -- 172.21.15.98:0/3982368203 learned_addr learned my addr 172.21.15.98:0/3982368203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f9801eda700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc100c90 0x7f97fc074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98026db700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 -- 172.21.15.98:0/3982368203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 msgr2=0x7f97fc0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 -- 172.21.15.98:0/3982368203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc100c90 msgr2=0x7f97fc074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc100c90 0x7f97fc074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98016d9700 1 -- 172.21.15.98:0/3982368203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80095f0 con 0x7f97fc101e80 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.989+0000 7f98026db700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f98016d9700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b530 tx=0x7f97f000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0011830 con 0x7f97fc101e80 2024-03-18T20:24:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f97f0011e20 con 0x7f97fc101e80 2024-03-18T20:24:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc1a0ec0 con 0x7f97fc101e80 2024-03-18T20:24:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0010dd0 con 0x7f97fc101e80 2024-03-18T20:24:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.990+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1a1390 con 0x7f97fc101e80 2024-03-18T20:24:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.991+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc108180 con 0x7f97fc101e80 2024-03-18T20:24:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.992+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f97f0011990 con 0x7f97fc101e80 2024-03-18T20:24:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.992+0000 7f97eeffd700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 0x7f97e80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.992+0000 7f9801eda700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 0x7f97e80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.992+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f97f00cc900 con 0x7f97fc101e80 2024-03-18T20:24:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.993+0000 7f9801eda700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 0x7f97e80a3660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f97f8000c00 tx=0x7f97f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:51.996+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97f0095ea0 con 0x7f97fc101e80 2024-03-18T20:24:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.202+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97fc067e60 con 0x7f97fc101e80 2024-03-18T20:24:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.204+0000 7f97eeffd700 1 -- 172.21.15.98:0/3982368203 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f97f0015090 con 0x7f97fc101e80 2024-03-18T20:24:52.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:52.208 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 msgr2=0x7f97e80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 0x7f97e80a3660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f97f8000c00 tx=0x7f97f8005fb0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 msgr2=0x7f97fc0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b530 tx=0x7f97f000b8f0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 shutdown_connections 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97fc102820 0x7f97fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f97e80a11a0 0x7f97e80a3660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97fc100c90 0x7f97fc074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 --2- 172.21.15.98:0/3982368203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97fc101e80 0x7f97fc0731a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 >> 172.21.15.98:0/3982368203 conn(0x7f97fc0fc200 msgr2=0x7f97fc1052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 shutdown_connections 2024-03-18T20:24:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.206+0000 7f980413e700 1 -- 172.21.15.98:0/3982368203 wait complete. 2024-03-18T20:24:52.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:24:52.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:24:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 -- 172.21.15.98:0/2120730329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 msgr2=0x7f6b74108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 --2- 172.21.15.98:0/2120730329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b74108eb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00b0a0 tx=0x7f6b6c00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 -- 172.21.15.98:0/2120730329 shutdown_connections 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 --2- 172.21.15.98:0/2120730329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b74108eb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 --2- 172.21.15.98:0/2120730329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 --2- 172.21.15.98:0/2120730329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b74104170 0x7f6b74106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 -- 172.21.15.98:0/2120730329 >> 172.21.15.98:0/2120730329 conn(0x7f6b740fb1d0 msgr2=0x7f6b740fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 -- 172.21.15.98:0/2120730329 shutdown_connections 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.343+0000 7f6b7af84700 1 -- 172.21.15.98:0/2120730329 wait complete. 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 Processor -- start 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 -- start start 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b74104170 0x7f6b74197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419e710 con 0x7f6b74106aa0 2024-03-18T20:24:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.344+0000 7f6b7af84700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419e850 con 0x7f6b74104170 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b7af84700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419e9e0 con 0x7f6b74101840 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45434/0 (socket says 172.21.15.98:45434) 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 -- 172.21.15.98:0/4293231883 learned_addr learned my addr 172.21.15.98:0/4293231883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b78d20700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 -- 172.21.15.98:0/4293231883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 msgr2=0x7f6b74196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 -- 172.21.15.98:0/4293231883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b74104170 msgr2=0x7f6b74197170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b74104170 0x7f6b74197170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 -- 172.21.15.98:0/4293231883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009d00 con 0x7f6b74106aa0 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b78d20700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b79521700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c012620 tx=0x7f6b6c012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00e040 con 0x7f6b74106aa0 2024-03-18T20:24:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b6c012be0 con 0x7f6b74106aa0 2024-03-18T20:24:52.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c01b6b0 con 0x7f6b74106aa0 2024-03-18T20:24:52.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.345+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7419ec60 con 0x7f6b74106aa0 2024-03-18T20:24:52.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.346+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7419f0e0 con 0x7f6b74106aa0 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.347+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b740fcd50 con 0x7f6b74106aa0 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.348+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6b6c012d50 con 0x7f6b74106aa0 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.348+0000 7f6b71ffb700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 0x7f6b640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.348+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6b6c0cefa0 con 0x7f6b74106aa0 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.348+0000 7f6b78d20700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 0x7f6b640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.349+0000 7f6b78d20700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 0x7f6b640a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b680099b0 tx=0x7f6b68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:52.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.352+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6b6c0985d0 con 0x7f6b74106aa0 2024-03-18T20:24:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.553+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b7402d130 con 0x7f6b74106aa0 2024-03-18T20:24:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.553+0000 7f6b71ffb700 1 -- 172.21.15.98:0/4293231883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b6c021020 con 0x7f6b74106aa0 2024-03-18T20:24:52.557 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:24:52.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 msgr2=0x7f6b640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 0x7f6b640a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b680099b0 tx=0x7f6b68008040 comp rx=0 tx=0).stop 2024-03-18T20:24:52.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 msgr2=0x7f6b7419e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c012620 tx=0x7f6b6c012700 comp rx=0 tx=0).stop 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 shutdown_connections 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b74106aa0 0x7f6b7419e140 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b74101840 0x7f6b74196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6b640a11a0 0x7f6b640a3660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 --2- 172.21.15.98:0/4293231883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b74104170 0x7f6b74197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 >> 172.21.15.98:0/4293231883 conn(0x7f6b740fb1d0 msgr2=0x7f6b74102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 shutdown_connections 2024-03-18T20:24:52.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.555+0000 7f6b7af84700 1 -- 172.21.15.98:0/4293231883 wait complete. 2024-03-18T20:24:52.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:52 smithi098 ceph-mon[43462]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 859 B/s wr, 2 op/s 2024-03-18T20:24:52.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:52 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3982368203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:52.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3333590073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 msgr2=0x7f6e841000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3333590073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e841000b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6e780099c0 tx=0x7f6e78009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3333590073 shutdown_connections 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3333590073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84101820 0x7f6e841080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3333590073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e841000b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3333590073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84100e80 0x7f6e841012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3333590073 >> 172.21.15.98:0/3333590073 conn(0x7f6e840fb200 msgr2=0x7f6e840fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3333590073 shutdown_connections 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3333590073 wait complete. 2024-03-18T20:24:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 Processor -- start 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- start start 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84100e80 0x7f6e84102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84101820 0x7f6e841027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84102ea0 con 0x7f6e840ffc90 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84103010 con 0x7f6e84101820 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.697+0000 7f6e8aed2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e841a0cf0 con 0x7f6e84100e80 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45444/0 (socket says 172.21.15.98:45444) 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 -- 172.21.15.98:0/3954053205 learned_addr learned my addr 172.21.15.98:0/3954053205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e8946f700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84101820 0x7f6e841027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e83fff700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84100e80 0x7f6e84102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 -- 172.21.15.98:0/3954053205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84100e80 msgr2=0x7f6e84102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84100e80 0x7f6e84102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 -- 172.21.15.98:0/3954053205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84101820 msgr2=0x7f6e841027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84101820 0x7f6e841027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 -- 172.21.15.98:0/3954053205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780095f0 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e88c6e700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e7800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7801d070 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e780054c0 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78004c50 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e841a0ef0 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.698+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841a1260 con 0x7f6e840ffc90 2024-03-18T20:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.699+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84067260 con 0x7f6e840ffc90 2024-03-18T20:24:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.700+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6e7800fcc0 con 0x7f6e840ffc90 2024-03-18T20:24:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.701+0000 7f6e81ffb700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 0x7f6e740a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.701+0000 7f6e83fff700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 0x7f6e740a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.701+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e780cd980 con 0x7f6e840ffc90 2024-03-18T20:24:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.701+0000 7f6e83fff700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 0x7f6e740a3660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009bf0 tx=0x7f6e70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.704+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e78096f50 con 0x7f6e840ffc90 2024-03-18T20:24:52.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:52 smithi136 ceph-mon[48073]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 859 B/s wr, 2 op/s 2024-03-18T20:24:52.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:52 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3982368203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:52.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:52 smithi169 ceph-mon[48529]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 859 B/s wr, 2 op/s 2024-03-18T20:24:52.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:52 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3982368203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:52.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.902+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6e841a1450 con 0x7f6e840ffc90 2024-03-18T20:24:52.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.902+0000 7f6e81ffb700 1 -- 172.21.15.98:0/3954053205 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6e780ba0a0 con 0x7f6e840ffc90 2024-03-18T20:24:52.906 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:24:52.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.904+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 msgr2=0x7f6e740a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.904+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 0x7f6e740a3660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009bf0 tx=0x7f6e70009260 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.904+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 msgr2=0x7f6e84103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.904+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e7800f740 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 shutdown_connections 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6e740a11a0 0x7f6e740a3660 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e84101820 0x7f6e841027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e840ffc90 0x7f6e84103b80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 --2- 172.21.15.98:0/3954053205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e84100e80 0x7f6e84102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 >> 172.21.15.98:0/3954053205 conn(0x7f6e840fb200 msgr2=0x7f6e84069000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:52.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 shutdown_connections 2024-03-18T20:24:52.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:52.905+0000 7f6e8aed2700 1 -- 172.21.15.98:0/3954053205 wait complete. 2024-03-18T20:24:52.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 -- 172.21.15.98:0/441630099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec101800 msgr2=0x7f60ec1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/441630099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec101800 0x7f60ec1080b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f60e400b0d0 tx=0x7f60e400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 -- 172.21.15.98:0/441630099 shutdown_connections 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/441630099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec101800 0x7f60ec1080b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/441630099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec0ffc70 0x7f60ec100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/441630099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec100e60 0x7f60ec1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 -- 172.21.15.98:0/441630099 >> 172.21.15.98:0/441630099 conn(0x7f60ec0fb200 msgr2=0x7f60ec0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 -- 172.21.15.98:0/441630099 shutdown_connections 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 -- 172.21.15.98:0/441630099 wait complete. 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.040+0000 7f60f2a2d700 1 Processor -- start 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 -- start start 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec0ffc70 0x7f60ec196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec101800 0x7f60ec19e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec19e800 con 0x7f60ec100e60 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec19e970 con 0x7f60ec0ffc70 2024-03-18T20:24:53.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f2a2d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec19eb30 con 0x7f60ec101800 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45472/0 (socket says 172.21.15.98:45472) 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 -- 172.21.15.98:0/3580810694 learned_addr learned my addr 172.21.15.98:0/3580810694 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60f0fca700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec101800 0x7f60ec19e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60ebfff700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec0ffc70 0x7f60ec196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 -- 172.21.15.98:0/3580810694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec101800 msgr2=0x7f60ec19e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec101800 0x7f60ec19e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 -- 172.21.15.98:0/3580810694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec0ffc70 msgr2=0x7f60ec196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec0ffc70 0x7f60ec196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 -- 172.21.15.98:0/3580810694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4009d00 con 0x7f60ec100e60 2024-03-18T20:24:53.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.041+0000 7f60eb7fe700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d950 tx=0x7f60e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.042+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e000f890 con 0x7f60ec100e60 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.042+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec19ee10 con 0x7f60ec100e60 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.042+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f60e000fe80 con 0x7f60ec100e60 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.042+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e000e3b0 con 0x7f60ec100e60 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.042+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec19f290 con 0x7f60ec100e60 2024-03-18T20:24:53.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.043+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec067f40 con 0x7f60ec100e60 2024-03-18T20:24:53.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.044+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f60e0010460 con 0x7f60ec100e60 2024-03-18T20:24:53.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.044+0000 7f60e97fa700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 0x7f60d40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.044+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f60e00ccbd0 con 0x7f60ec100e60 2024-03-18T20:24:53.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.044+0000 7f60ebfff700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 0x7f60d40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.045+0000 7f60ebfff700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 0x7f60d40a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0fd250 tx=0x7f60dc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.048+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60e0096170 con 0x7f60ec100e60 2024-03-18T20:24:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.241+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f60ec19f480 con 0x7f60ec100e60 2024-03-18T20:24:53.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.242+0000 7f60e97fa700 1 -- 172.21.15.98:0/3580810694 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60e00200a0 con 0x7f60ec100e60 2024-03-18T20:24:53.247 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:24:53.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 msgr2=0x7f60d40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 0x7f60d40a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0fd250 tx=0x7f60dc00a380 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 msgr2=0x7f60ec197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d950 tx=0x7f60e000dc60 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 shutdown_connections 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60ec101800 0x7f60ec19e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f60d40a1120 0x7f60d40a35e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60ec0ffc70 0x7f60ec196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 --2- 172.21.15.98:0/3580810694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ec100e60 0x7f60ec197220 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 >> 172.21.15.98:0/3580810694 conn(0x7f60ec0fb200 msgr2=0x7f60ec069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 shutdown_connections 2024-03-18T20:24:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.244+0000 7f60f2a2d700 1 -- 172.21.15.98:0/3580810694 wait complete. 2024-03-18T20:24:53.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:24:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 -- 172.21.15.98:0/4256055333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb80ffc70 msgr2=0x7f7fb8100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 --2- 172.21.15.98:0/4256055333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8100090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40099c0 tx=0x7f7fb4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 -- 172.21.15.98:0/4256055333 shutdown_connections 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 --2- 172.21.15.98:0/4256055333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb8101800 0x7f7fb81080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 --2- 172.21.15.98:0/4256055333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8100090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 --2- 172.21.15.98:0/4256055333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb8100e60 0x7f7fb81012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.381+0000 7f7fbf980700 1 -- 172.21.15.98:0/4256055333 >> 172.21.15.98:0/4256055333 conn(0x7f7fb80fb200 msgr2=0x7f7fb80fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- 172.21.15.98:0/4256055333 shutdown_connections 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- 172.21.15.98:0/4256055333 wait complete. 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 Processor -- start 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- start start 2024-03-18T20:24:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb8101800 0x7f7fb819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819e6d0 con 0x7f7fb8100e60 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819e810 con 0x7f7fb8101800 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbf980700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819e9a0 con 0x7f7fb80ffc70 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbcf1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbcf1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60112/0 (socket says 172.21.15.98:60112) 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.382+0000 7f7fbcf1b700 1 -- 172.21.15.98:0/1942481579 learned_addr learned my addr 172.21.15.98:0/1942481579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbd71c700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 -- 172.21.15.98:0/1942481579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb80ffc70 msgr2=0x7f7fb8196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 -- 172.21.15.98:0/1942481579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb8101800 msgr2=0x7f7fb819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb8101800 0x7f7fb819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 -- 172.21.15.98:0/1942481579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40095f0 con 0x7f7fb8100e60 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbcf1b700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800d950 tx=0x7f7fa800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8009b40 con 0x7f7fb8100e60 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7fa8009ca0 con 0x7f7fb8100e60 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb819ec80 con 0x7f7fb8100e60 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa800f620 con 0x7f7fb8100e60 2024-03-18T20:24:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.383+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb819f190 con 0x7f7fb8100e60 2024-03-18T20:24:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.384+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb8067260 con 0x7f7fb8100e60 2024-03-18T20:24:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.385+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7fa8010460 con 0x7f7fb8100e60 2024-03-18T20:24:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.385+0000 7f7fae7fc700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 0x7f7fa40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.386+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7fa80cce30 con 0x7f7fb8100e60 2024-03-18T20:24:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.386+0000 7f7fbd71c700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 0x7f7fa40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.386+0000 7f7fbd71c700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 0x7f7fa40a36e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.390+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7fa80d0030 con 0x7f7fb8100e60 2024-03-18T20:24:53.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:53 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4293231883' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:53 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3954053205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:53 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3580810694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.592+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7fb819f380 con 0x7f7fb8100e60 2024-03-18T20:24:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.592+0000 7f7fae7fc700 1 -- 172.21.15.98:0/1942481579 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7fa80200a0 con 0x7f7fb8100e60 2024-03-18T20:24:53.599 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:24:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 msgr2=0x7f7fa40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 0x7f7fa40a36e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4005fd0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 msgr2=0x7f7fb81970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800d950 tx=0x7f7fa800dd10 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 shutdown_connections 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7fa40a1220 0x7f7fa40a36e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb8101800 0x7f7fb819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb80ffc70 0x7f7fb8196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 --2- 172.21.15.98:0/1942481579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fb8100e60 0x7f7fb81970e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 >> 172.21.15.98:0/1942481579 conn(0x7f7fb80fb200 msgr2=0x7f7fb8069020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 shutdown_connections 2024-03-18T20:24:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.594+0000 7f7fbf980700 1 -- 172.21.15.98:0/1942481579 wait complete. 2024-03-18T20:24:53.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- 172.21.15.98:0/470189047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 msgr2=0x7f7f1c100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 --2- 172.21.15.98:0/470189047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c100090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0099c0 tx=0x7f7f0c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- 172.21.15.98:0/470189047 shutdown_connections 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 --2- 172.21.15.98:0/470189047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 0x7f7f1c1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 --2- 172.21.15.98:0/470189047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c100090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 --2- 172.21.15.98:0/470189047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 0x7f7f1c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- 172.21.15.98:0/470189047 >> 172.21.15.98:0/470189047 conn(0x7f7f1c0fb200 msgr2=0x7f7f1c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- 172.21.15.98:0/470189047 shutdown_connections 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- 172.21.15.98:0/470189047 wait complete. 2024-03-18T20:24:53.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 Processor -- start 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.730+0000 7f7f241df700 1 -- start start 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 0x7f7f1c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 0x7f7f1c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c19e7d0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c19e940 con 0x7f7f1c101800 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c19eb00 con 0x7f7f1c100e60 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60128/0 (socket says 172.21.15.98:60128) 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 -- 172.21.15.98:0/1215725107 learned_addr learned my addr 172.21.15.98:0/1215725107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f2177a700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 0x7f7f1c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f2277c700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 0x7f7f1c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 -- 172.21.15.98:0/1215725107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 msgr2=0x7f7f1c19ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 0x7f7f1c19ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 -- 172.21.15.98:0/1215725107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 msgr2=0x7f7f1c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 0x7f7f1c19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 -- 172.21.15.98:0/1215725107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c0095f0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f21f7b700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009f50 tx=0x7f7f0c005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c01d070 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7f0c00bda0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c022d30 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c19ed80 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.731+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c19f150 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.732+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c1071e0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.733+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7f0c022500 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.734+0000 7f7f12ffd700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 0x7f7f080a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.734+0000 7f7f2177a700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 0x7f7f080a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.734+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7f0c0cde90 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.734+0000 7f7f2177a700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 0x7f7f080a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f18004630 tx=0x7f7f1800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:53.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:53 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4293231883' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:53 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3954053205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:53 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3580810694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.738+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f0c0974c0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:53 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4293231883' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:53 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3954053205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:53 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3580810694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.961+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7f1c067260 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.961+0000 7f7f12ffd700 1 -- 172.21.15.98:0/1215725107 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+590 (secure 0 0 0) 0x7f7f0c09aae0 con 0x7f7f1c0ffc70 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2681086384 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3671856112 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1417871753 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2768304991 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2082705134 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:53.967 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3238391768 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 msgr2=0x7f7f080a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 0x7f7f080a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f18004630 tx=0x7f7f1800b3c0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 msgr2=0x7f7f1c069ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009f50 tx=0x7f7f0c005b00 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 shutdown_connections 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f080a1120 0x7f7f080a35e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f1c101800 0x7f7f1c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c0ffc70 0x7f7f1c069ae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 --2- 172.21.15.98:0/1215725107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f1c100e60 0x7f7f1c19ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 >> 172.21.15.98:0/1215725107 conn(0x7f7f1c0fb200 msgr2=0x7f7f1c104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 shutdown_connections 2024-03-18T20:24:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:53.963+0000 7f7f241df700 1 -- 172.21.15.98:0/1215725107 wait complete. 2024-03-18T20:24:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-18T20:24:53.980 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:24:53.980 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:24:53.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:54.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- 172.21.15.98:0/3653340653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 msgr2=0x7f7b04108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 --2- 172.21.15.98:0/3653340653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b04108e00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00b0a0 tx=0x7f7afc00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- 172.21.15.98:0/3653340653 shutdown_connections 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 --2- 172.21.15.98:0/3653340653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b04108e00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 --2- 172.21.15.98:0/3653340653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 0x7f7b04103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 --2- 172.21.15.98:0/3653340653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 0x7f7b041064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- 172.21.15.98:0/3653340653 >> 172.21.15.98:0/3653340653 conn(0x7f7b040fb100 msgr2=0x7f7b040fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- 172.21.15.98:0/3653340653 shutdown_connections 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- 172.21.15.98:0/3653340653 wait complete. 2024-03-18T20:24:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 Processor -- start 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.156+0000 7f7b0979e700 1 -- start start 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 0x7f7b04101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 0x7f7b040ff760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b041003a0 con 0x7f7b04106a10 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04100510 con 0x7f7b041017b0 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b041a0bc0 con 0x7f7b041040e0 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60144/0 (socket says 172.21.15.98:60144) 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 -- 172.21.15.98:0/1379167393 learned_addr learned my addr 172.21.15.98:0/1379167393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b02ffd700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 0x7f7b04101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b027fc700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 0x7f7b040ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 -- 172.21.15.98:0/1379167393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 msgr2=0x7f7b040ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 0x7f7b040ff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 -- 172.21.15.98:0/1379167393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 msgr2=0x7f7b04101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 0x7f7b04101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 -- 172.21.15.98:0/1379167393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc009d00 con 0x7f7b04106a10 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b037fe700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009820 tx=0x7f7afc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc00e040 con 0x7f7b04106a10 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7afc004c50 con 0x7f7b04106a10 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b041a0dc0 con 0x7f7b04106a10 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0052a0 con 0x7f7b04106a10 2024-03-18T20:24:54.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.157+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b041a1230 con 0x7f7b04106a10 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.158+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b040fcc80 con 0x7f7b04106a10 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.159+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7afc005400 con 0x7f7b04106a10 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.160+0000 7f7aebfff700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 0x7f7aec0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.160+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7afc0cdfc0 con 0x7f7b04106a10 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.160+0000 7f7b02ffd700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 0x7f7aec0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.160+0000 7f7b02ffd700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 0x7f7aec0a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7af800acd0 tx=0x7f7af800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.164+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7afc1019f0 con 0x7f7b04106a10 2024-03-18T20:24:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.374+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b04067e60 con 0x7f7b04106a10 2024-03-18T20:24:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.375+0000 7f7aebfff700 1 -- 172.21.15.98:0/1379167393 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f7afc0975f0 con 0x7f7b04106a10 2024-03-18T20:24:54.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:54.381 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 msgr2=0x7f7aec0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 0x7f7aec0a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7af800acd0 tx=0x7f7af800a380 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 msgr2=0x7f7b040ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009820 tx=0x7f7afc00bab0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 shutdown_connections 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b04106a10 0x7f7b040ffca0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7aec0a11a0 0x7f7aec0a3660 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b041017b0 0x7f7b04101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 --2- 172.21.15.98:0/1379167393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b041040e0 0x7f7b040ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 >> 172.21.15.98:0/1379167393 conn(0x7f7b040fb100 msgr2=0x7f7b041021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 shutdown_connections 2024-03-18T20:24:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.378+0000 7f7b0979e700 1 -- 172.21.15.98:0/1379167393 wait complete. 2024-03-18T20:24:54.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:24:54.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/2337440489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec0ffc70 msgr2=0x7f8cec100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/2337440489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec0ffc70 0x7f8cec100090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8cd40099c0 tx=0x7f8cd4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/2337440489 shutdown_connections 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/2337440489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 0x7f8cec1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/2337440489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec0ffc70 0x7f8cec100090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/2337440489 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec100e60 0x7f8cec1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/2337440489 >> 172.21.15.98:0/2337440489 conn(0x7f8cec0fb200 msgr2=0x7f8cec0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/2337440489 shutdown_connections 2024-03-18T20:24:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.517+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/2337440489 wait complete. 2024-03-18T20:24:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 Processor -- start 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 -- start start 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec0ffc70 0x7f8cec069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 0x7f8cec19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec19e6e0 con 0x7f8cec100e60 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec19e850 con 0x7f8cec101800 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cf0ca3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec19ea10 con 0x7f8cec0ffc70 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60160/0 (socket says 172.21.15.98:60160) 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 -- 172.21.15.98:0/600432941 learned_addr learned my addr 172.21.15.98:0/600432941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cea59c700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec0ffc70 0x7f8cec069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8cead9d700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 0x7f8cec19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 -- 172.21.15.98:0/600432941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec0ffc70 msgr2=0x7f8cec069ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec0ffc70 0x7f8cec069ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 -- 172.21.15.98:0/600432941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 msgr2=0x7f8cec19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 0x7f8cec19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.518+0000 7f8ce9d9b700 1 -- 172.21.15.98:0/600432941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40095f0 con 0x7f8cec100e60 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8ce9d9b700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00d950 tx=0x7f8cdc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc00f890 con 0x7f8cec100e60 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec19ecf0 con 0x7f8cec100e60 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8cdc00fe80 con 0x7f8cec100e60 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec19f1f0 con 0x7f8cec100e60 2024-03-18T20:24:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.519+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc01b450 con 0x7f8cec100e60 2024-03-18T20:24:54.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.520+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec1071e0 con 0x7f8cec100e60 2024-03-18T20:24:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.521+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8cdc010460 con 0x7f8cec100e60 2024-03-18T20:24:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.521+0000 7f8ce37fe700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 0x7f8cd80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.521+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8cdc0ccbf0 con 0x7f8cec100e60 2024-03-18T20:24:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.522+0000 7f8cea59c700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 0x7f8cd80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.522+0000 7f8cea59c700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 0x7f8cd80a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4006010 tx=0x7f8cd4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.525+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8cdc096190 con 0x7f8cec100e60 2024-03-18T20:24:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.720+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8cec02d130 con 0x7f8cec100e60 2024-03-18T20:24:54.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.721+0000 7f8ce37fe700 1 -- 172.21.15.98:0/600432941 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8cdc016020 con 0x7f8cec100e60 2024-03-18T20:24:54.725 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 msgr2=0x7f8cd80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 0x7f8cd80a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4006010 tx=0x7f8cd4000bc0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 msgr2=0x7f8cec19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00d950 tx=0x7f8cdc00dc60 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 shutdown_connections 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8cd80a1120 0x7f8cd80a35e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8cec101800 0x7f8cec19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cec0ffc70 0x7f8cec069ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 --2- 172.21.15.98:0/600432941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8cec100e60 0x7f8cec19b990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 >> 172.21.15.98:0/600432941 conn(0x7f8cec0fb200 msgr2=0x7f8cec104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 shutdown_connections 2024-03-18T20:24:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.723+0000 7f8cf0ca3700 1 -- 172.21.15.98:0/600432941 wait complete. 2024-03-18T20:24:54.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:54 smithi098 ceph-mon[43462]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-18T20:24:54.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:54 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1942481579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:54.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:54 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1215725107' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:54.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:54 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1379167393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:54.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:24:54.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:54 smithi136 ceph-mon[48073]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-18T20:24:54.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:54 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1942481579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:54.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:54 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1215725107' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:54.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:54 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1379167393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.860+0000 7f4d0e51e700 1 -- 172.21.15.98:0/3448964631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 msgr2=0x7f4d080ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.860+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/3448964631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d080ff460 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009990 tx=0x7f4cf8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.860+0000 7f4d0e51e700 1 -- 172.21.15.98:0/3448964631 shutdown_connections 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.860+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/3448964631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d081029b0 0x7f4d08109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/3448964631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d080ff460 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/3448964631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d08102010 0x7f4d08102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 -- 172.21.15.98:0/3448964631 >> 172.21.15.98:0/3448964631 conn(0x7f4d080fa990 msgr2=0x7f4d080fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 -- 172.21.15.98:0/3448964631 shutdown_connections 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 -- 172.21.15.98:0/3448964631 wait complete. 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 Processor -- start 2024-03-18T20:24:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.861+0000 7f4d0e51e700 1 -- start start 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 0x7f4d081971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 0x7f4d0819e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0819e770 con 0x7f4d080ff040 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0819e8e0 con 0x7f4d081029b0 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0819eaa0 con 0x7f4d08102010 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60164/0 (socket says 172.21.15.98:60164) 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 -- 172.21.15.98:0/2627677318 learned_addr learned my addr 172.21.15.98:0/2627677318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:54.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0cabb700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 0x7f4d0819e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d077fe700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 0x7f4d081971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 -- 172.21.15.98:0/2627677318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 msgr2=0x7f4d081971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 0x7f4d081971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 -- 172.21.15.98:0/2627677318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 msgr2=0x7f4d0819e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 0x7f4d0819e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 -- 172.21.15.98:0/2627677318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80095f0 con 0x7f4d080ff040 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d077fe700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 0x7f4d081971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0cabb700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 0x7f4d0819e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d07fff700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80098e0 tx=0x7f4cf8003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf801d070 con 0x7f4d080ff040 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4cf800bd90 con 0x7f4d080ff040 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0819ed20 con 0x7f4d080ff040 2024-03-18T20:24:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8022dc0 con 0x7f4d080ff040 2024-03-18T20:24:54.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.862+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0819f140 con 0x7f4d080ff040 2024-03-18T20:24:54.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.863+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d08108290 con 0x7f4d080ff040 2024-03-18T20:24:54.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.864+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4cf8003ff0 con 0x7f4d080ff040 2024-03-18T20:24:54.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.865+0000 7f4d057fa700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 0x7f4cf00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:54.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.865+0000 7f4d077fe700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 0x7f4cf00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:54.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.865+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4cf80ce080 con 0x7f4d080ff040 2024-03-18T20:24:54.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.865+0000 7f4d077fe700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 0x7f4cf00a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc005fd0 tx=0x7f4cfc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:54.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:54.869+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4cf8097650 con 0x7f4d080ff040 2024-03-18T20:24:54.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:54 smithi169 ceph-mon[48529]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-18T20:24:54.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:54 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1942481579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:54.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:54 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1215725107' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:54.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:54 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1379167393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.064+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d0802cee0 con 0x7f4d080ff040 2024-03-18T20:24:55.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.065+0000 7f4d057fa700 1 -- 172.21.15.98:0/2627677318 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4cf809ac70 con 0x7f4d080ff040 2024-03-18T20:24:55.069 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:24:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 msgr2=0x7f4cf00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 0x7f4cf00a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc005fd0 tx=0x7f4cfc005ee0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 msgr2=0x7f4d08196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80098e0 tx=0x7f4cf8003b00 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 shutdown_connections 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4cf00a1120 0x7f4cf00a35e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d081029b0 0x7f4d0819e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d080ff040 0x7f4d08196c80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 --2- 172.21.15.98:0/2627677318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d08102010 0x7f4d081971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 >> 172.21.15.98:0/2627677318 conn(0x7f4d080fa990 msgr2=0x7f4d081053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 shutdown_connections 2024-03-18T20:24:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.067+0000 7f4d0e51e700 1 -- 172.21.15.98:0/2627677318 wait complete. 2024-03-18T20:24:55.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 -- 172.21.15.98:0/1099591395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9141017e0 msgr2=0x7fd914103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1099591395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9141017e0 0x7fd914103bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd908009990 tx=0x7fd908009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 -- 172.21.15.98:0/1099591395 shutdown_connections 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1099591395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 0x7fd914108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1099591395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9141017e0 0x7fd914103bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1099591395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd914104110 0x7fd914106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 -- 172.21.15.98:0/1099591395 >> 172.21.15.98:0/1099591395 conn(0x7fd9140fb1d0 msgr2=0x7fd9140fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 -- 172.21.15.98:0/1099591395 shutdown_connections 2024-03-18T20:24:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.204+0000 7fd918e7c700 1 -- 172.21.15.98:0/1099591395 wait complete. 2024-03-18T20:24:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 Processor -- start 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 -- start start 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9141017e0 0x7fd914196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 0x7fd91419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91419e720 con 0x7fd914104110 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91419e860 con 0x7fd9141017e0 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd918e7c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91419e9f0 con 0x7fd914106a40 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60168/0 (socket says 172.21.15.98:60168) 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 -- 172.21.15.98:0/1124925898 learned_addr learned my addr 172.21.15.98:0/1124925898 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd91259c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9141017e0 0x7fd914196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd912d9d700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 0x7fd91419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 -- 172.21.15.98:0/1124925898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 msgr2=0x7fd91419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 0x7fd91419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 -- 172.21.15.98:0/1124925898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9141017e0 msgr2=0x7fd914196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9141017e0 0x7fd914196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 -- 172.21.15.98:0/1124925898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080095f0 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.205+0000 7fd911d9b700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd90400d950 tx=0x7fd90400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.206+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400f890 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.206+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd90400fe80 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.206+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91419ecd0 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.206+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400e3b0 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.206+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91419f1d0 con 0x7fd914104110 2024-03-18T20:24:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.207+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9140fcd50 con 0x7fd914104110 2024-03-18T20:24:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.208+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd904010460 con 0x7fd914104110 2024-03-18T20:24:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.208+0000 7fd9037fe700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 0x7fd8fc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.208+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd9040cc7f0 con 0x7fd914104110 2024-03-18T20:24:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.208+0000 7fd91259c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 0x7fd8fc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.209+0000 7fd91259c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 0x7fd8fc0a3660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd908000c00 tx=0x7fd908005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.212+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd904095d90 con 0x7fd914104110 2024-03-18T20:24:55.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.409+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd91402d130 con 0x7fd914104110 2024-03-18T20:24:55.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.410+0000 7fd9037fe700 1 -- 172.21.15.98:0/1124925898 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd91402d130 con 0x7fd914104110 2024-03-18T20:24:55.415 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 msgr2=0x7fd8fc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 0x7fd8fc0a3660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd908000c00 tx=0x7fd908005fb0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 msgr2=0x7fd914197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd90400d950 tx=0x7fd90400dc60 comp rx=0 tx=0).stop 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 shutdown_connections 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd914106a40 0x7fd91419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd8fc0a11a0 0x7fd8fc0a3660 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9141017e0 0x7fd914196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 --2- 172.21.15.98:0/1124925898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd914104110 0x7fd914197130 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 >> 172.21.15.98:0/1124925898 conn(0x7fd9140fb1d0 msgr2=0x7fd9141022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 shutdown_connections 2024-03-18T20:24:55.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.412+0000 7fd918e7c700 1 -- 172.21.15.98:0/1124925898 wait complete. 2024-03-18T20:24:55.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:24:55.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 -- 172.21.15.98:0/3740257411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac1017e0 msgr2=0x7f4dac103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/3740257411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac1017e0 0x7f4dac103bd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009990 tx=0x7f4d9c009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 -- 172.21.15.98:0/3740257411 shutdown_connections 2024-03-18T20:24:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/3740257411 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 0x7f4dac108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/3740257411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac1017e0 0x7f4dac103bd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/3740257411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac104110 0x7f4dac106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 -- 172.21.15.98:0/3740257411 >> 172.21.15.98:0/3740257411 conn(0x7f4dac0fb1d0 msgr2=0x7f4dac0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 -- 172.21.15.98:0/3740257411 shutdown_connections 2024-03-18T20:24:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.549+0000 7f4db2a2b700 1 -- 172.21.15.98:0/3740257411 wait complete. 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 Processor -- start 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 -- start start 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac1017e0 0x7f4dac101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 0x7f4dac0ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac100470 con 0x7f4dac104110 2024-03-18T20:24:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1005e0 con 0x7f4dac1017e0 2024-03-18T20:24:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db2a2b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1a0c50 con 0x7f4dac106a40 2024-03-18T20:24:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60184/0 (socket says 172.21.15.98:60184) 2024-03-18T20:24:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 -- 172.21.15.98:0/88756930 learned_addr learned my addr 172.21.15.98:0/88756930 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dabfff700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac1017e0 0x7f4dac101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4db0fc8700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 0x7f4dac0ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 -- 172.21.15.98:0/88756930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 msgr2=0x7f4dac0ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 0x7f4dac0ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 -- 172.21.15.98:0/88756930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac1017e0 msgr2=0x7f4dac101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac1017e0 0x7f4dac101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.550+0000 7f4dab7fe700 1 -- 172.21.15.98:0/88756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0095f0 con 0x7f4dac104110 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4dab7fe700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4da000e9a0 tx=0x7f4da000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0009790 con 0x7f4dac104110 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac1a0ee0 con 0x7f4dac104110 2024-03-18T20:24:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4da0010430 con 0x7f4dac104110 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0018680 con 0x7f4dac104110 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.551+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac1a13b0 con 0x7f4dac104110 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.552+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac0fcd50 con 0x7f4dac104110 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.553+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4da00105a0 con 0x7f4dac104110 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.553+0000 7f4da97fa700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 0x7f4d940a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.553+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4da0014070 con 0x7f4dac104110 2024-03-18T20:24:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.553+0000 7f4dabfff700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 0x7f4d940a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.554+0000 7f4dabfff700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 0x7f4d940a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006010 tx=0x7f4d9c00b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.557+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4da0096420 con 0x7f4dac104110 2024-03-18T20:24:55.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:55 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/600432941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:55 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2627677318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:55 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1124925898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.753+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4dac02d130 con 0x7f4dac104110 2024-03-18T20:24:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.753+0000 7f4da97fa700 1 -- 172.21.15.98:0/88756930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4da0026020 con 0x7f4dac104110 2024-03-18T20:24:55.758 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:24:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.755+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 msgr2=0x7f4d940a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.755+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 0x7f4d940a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006010 tx=0x7f4d9c00b610 comp rx=0 tx=0).stop 2024-03-18T20:24:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 msgr2=0x7f4dac0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4da000e9a0 tx=0x7f4da000ecb0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 shutdown_connections 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dac106a40 0x7f4dac0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4d940a1120 0x7f4d940a35e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4dac1017e0 0x7f4dac101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 --2- 172.21.15.98:0/88756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac104110 0x7f4dac0ff790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 >> 172.21.15.98:0/88756930 conn(0x7f4dac0fb1d0 msgr2=0x7f4dac103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 shutdown_connections 2024-03-18T20:24:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.756+0000 7f4db2a2b700 1 -- 172.21.15.98:0/88756930 wait complete. 2024-03-18T20:24:55.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:55 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/600432941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:55 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2627677318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:55 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1124925898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:24:55.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:55 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/600432941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:55 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2627677318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:55 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1124925898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:24:55.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 -- 172.21.15.98:0/1592827137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 msgr2=0x7fb8541012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 --2- 172.21.15.98:0/1592827137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb8541012b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb8440099c0 tx=0x7fb844009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 -- 172.21.15.98:0/1592827137 shutdown_connections 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 --2- 172.21.15.98:0/1592827137 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8541017f0 0x7fb8541080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 --2- 172.21.15.98:0/1592827137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 0x7fb854100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 --2- 172.21.15.98:0/1592827137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb8541012b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 -- 172.21.15.98:0/1592827137 >> 172.21.15.98:0/1592827137 conn(0x7fb8540fb1d0 msgr2=0x7fb8540fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 -- 172.21.15.98:0/1592827137 shutdown_connections 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 -- 172.21.15.98:0/1592827137 wait complete. 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.891+0000 7fb859efa700 1 Processor -- start 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 -- start start 2024-03-18T20:24:55.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 0x7fb854068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8541017f0 0x7fb85419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85419e730 con 0x7fb854100e50 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85419e8a0 con 0x7fb8540ffc60 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85419ea60 con 0x7fb8541017f0 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60196/0 (socket says 172.21.15.98:60196) 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 -- 172.21.15.98:0/1545753371 learned_addr learned my addr 172.21.15.98:0/1545753371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb8537fe700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 0x7fb854068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 -- 172.21.15.98:0/1545753371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8541017f0 msgr2=0x7fb85419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8541017f0 0x7fb85419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 -- 172.21.15.98:0/1545753371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 msgr2=0x7fb854068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:55.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 0x7fb854068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 -- 172.21.15.98:0/1545753371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440095f0 con 0x7fb854100e50 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb852ffd700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb844005910 tx=0x7fb844005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84401d070 con 0x7fb854100e50 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85419ece0 con 0x7fb854100e50 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb84400bd30 con 0x7fb854100e50 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.892+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844022d80 con 0x7fb854100e50 2024-03-18T20:24:55.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.893+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85419f100 con 0x7fb854100e50 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.894+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb854107120 con 0x7fb854100e50 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.894+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb844022470 con 0x7fb854100e50 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.895+0000 7fb850ff9700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 0x7fb8400a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.895+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb8440cd9a0 con 0x7fb854100e50 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.895+0000 7fb8537fe700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 0x7fb8400a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.895+0000 7fb8537fe700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 0x7fb8400a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb854100cb0 tx=0x7fb83c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:55.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:55.899+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb844096fd0 con 0x7fb854100e50 2024-03-18T20:24:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.121+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb854067af0 con 0x7fb854100e50 2024-03-18T20:24:56.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.121+0000 7fb850ff9700 1 -- 172.21.15.98:0/1545753371 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+590 (secure 0 0 0) 0x7fb84409a5f0 con 0x7fb854100e50 2024-03-18T20:24:56.126 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:56.126 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2681086384 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3671856112 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1417871753 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2768304991 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2082705134 2024-03-19T20:19:45.512552+0000 2024-03-18T20:24:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3238391768 2024-03-19T20:19:25.585401+0000 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 msgr2=0x7fb8400a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 0x7fb8400a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb854100cb0 tx=0x7fb83c00b410 comp rx=0 tx=0).stop 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 msgr2=0x7fb85419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb844005910 tx=0x7fb844005b00 comp rx=0 tx=0).stop 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 shutdown_connections 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8541017f0 0x7fb85419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb8400a1120 0x7fb8400a35e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8540ffc60 0x7fb854068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 --2- 172.21.15.98:0/1545753371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb854100e50 0x7fb85419b9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.123+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 >> 172.21.15.98:0/1545753371 conn(0x7fb8540fb1d0 msgr2=0x7fb854104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.124+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 shutdown_connections 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.124+0000 7fb859efa700 1 -- 172.21.15.98:0/1545753371 wait complete. 2024-03-18T20:24:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-18T20:24:56.140 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-03-18T20:24:56.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:56.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T20:24:56.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[97242]: fuse finished with error 0 and tester_r 0 2024-03-18T20:24:56.180 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:24:56.275 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:24:56.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:56.275 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T20:24:56.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.468+0000 7f48bb2b4700 1 -- 172.21.15.98:0/3817046031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b4100d50 msgr2=0x7f48b41011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.468+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/3817046031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b4100d50 0x7f48b41011b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f48a40099c0 tx=0x7f48a4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.468+0000 7f48bb2b4700 1 -- 172.21.15.98:0/3817046031 shutdown_connections 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.468+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/3817046031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b41016f0 0x7f48b4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.468+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/3817046031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 0x7f48b40fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/3817046031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b4100d50 0x7f48b41011b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- 172.21.15.98:0/3817046031 >> 172.21.15.98:0/3817046031 conn(0x7f48b40fb0d0 msgr2=0x7f48b40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- 172.21.15.98:0/3817046031 shutdown_connections 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- 172.21.15.98:0/3817046031 wait complete. 2024-03-18T20:24:56.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 Processor -- start 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- start start 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 0x7f48b4195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b4100d50 0x7f48b4195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d490 con 0x7f48b41016f0 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d5d0 con 0x7f48b40ffb60 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48bb2b4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d760 con 0x7f48b4100d50 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48b9851700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48b9851700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60216/0 (socket says 172.21.15.98:60216) 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.469+0000 7f48b9851700 1 -- 172.21.15.98:0/1713242129 learned_addr learned my addr 172.21.15.98:0/1713242129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b884f700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b4100d50 0x7f48b4195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9050700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 0x7f48b4195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 -- 172.21.15.98:0/1713242129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b4100d50 msgr2=0x7f48b4195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b4100d50 0x7f48b4195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 -- 172.21.15.98:0/1713242129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 msgr2=0x7f48b4195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 0x7f48b4195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 -- 172.21.15.98:0/1713242129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48b9851700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007ee0 tx=0x7f48ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac017070 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f48ac00f040 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b419da40 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac005570 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.470+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b419df00 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.471+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b4106fe0 con 0x7f48b41016f0 2024-03-18T20:24:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.472+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f48ac00db20 con 0x7f48b41016f0 2024-03-18T20:24:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.473+0000 7f48aa7fc700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.473+0000 7f48b9050700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.473+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f48ac0cc8b0 con 0x7f48b41016f0 2024-03-18T20:24:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.473+0000 7f48b9050700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48b4100bb0 tx=0x7f48b000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.476+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f48ac095e50 con 0x7f48b41016f0 2024-03-18T20:24:56.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.680+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48b4067a10 con 0x7f48b41016f0 2024-03-18T20:24:56.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.682+0000 7f48aa7fc700 1 -- 172.21.15.98:0/1713242129 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f48ac099470 con 0x7f48b41016f0 2024-03-18T20:24:56.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:56.687 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:56.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 msgr2=0x7f48a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48b4100bb0 tx=0x7f48b000a3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 msgr2=0x7f48b419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007ee0 tx=0x7f48ac00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 shutdown_connections 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41016f0 0x7f48b419cec0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f48a00a1120 0x7f48a00a35e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48b40ffb60 0x7f48b4195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 --2- 172.21.15.98:0/1713242129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f48b4100d50 0x7f48b4195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 >> 172.21.15.98:0/1713242129 conn(0x7f48b40fb0d0 msgr2=0x7f48b4104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 shutdown_connections 2024-03-18T20:24:56.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.684+0000 7f48bb2b4700 1 -- 172.21.15.98:0/1713242129 wait complete. 2024-03-18T20:24:56.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:56 smithi098 ceph-mon[43462]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-18T20:24:56.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:56 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/88756930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:56.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:56 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1545753371' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:56.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:24:56.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:56 smithi136 ceph-mon[48073]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-18T20:24:56.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:56 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/88756930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:56.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:56 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1545753371' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 -- 172.21.15.98:0/310076790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed4100d80 msgr2=0x7fbed41011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 --2- 172.21.15.98:0/310076790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed4100d80 0x7fbed41011e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbec40099c0 tx=0x7fbec4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 -- 172.21.15.98:0/310076790 shutdown_connections 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 --2- 172.21.15.98:0/310076790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 0x7fbed4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 --2- 172.21.15.98:0/310076790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed40ffb90 0x7fbed40fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 --2- 172.21.15.98:0/310076790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed4100d80 0x7fbed41011e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 -- 172.21.15.98:0/310076790 >> 172.21.15.98:0/310076790 conn(0x7fbed40fb100 msgr2=0x7fbed40fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 -- 172.21.15.98:0/310076790 shutdown_connections 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.819+0000 7fbedc0be700 1 -- 172.21.15.98:0/310076790 wait complete. 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 Processor -- start 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 -- start start 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed4100d80 0x7fbed419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 0x7fbed41a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed41a0840 con 0x7fbed40ffb90 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed41a09b0 con 0x7fbed4100d80 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbedc0be700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed41a0b70 con 0x7fbed4101720 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60222/0 (socket says 172.21.15.98:60222) 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 -- 172.21.15.98:0/1001287803 learned_addr learned my addr 172.21.15.98:0/1001287803 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9659700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed4100d80 0x7fbed419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbeda65b700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 0x7fbed41a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 -- 172.21.15.98:0/1001287803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 msgr2=0x7fbed41a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 0x7fbed41a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 -- 172.21.15.98:0/1001287803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed4100d80 msgr2=0x7fbed419dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed4100d80 0x7fbed419dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 -- 172.21.15.98:0/1001287803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec40095f0 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.820+0000 7fbed9e5a700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbed000d980 tx=0x7fbed000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0004020 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbed0009730 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed41a0e50 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0003a00 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed41a12d0 con 0x7fbed40ffb90 2024-03-18T20:24:56.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.821+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed41070b0 con 0x7fbed40ffb90 2024-03-18T20:24:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.823+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fbed00098a0 con 0x7fbed40ffb90 2024-03-18T20:24:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.823+0000 7fbecaffd700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.823+0000 7fbed9659700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:56.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.823+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbed00ccaf0 con 0x7fbed40ffb90 2024-03-18T20:24:56.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.824+0000 7fbed9659700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 0x7fbec00a3660 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbec400c010 tx=0x7fbec4003720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:56.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:56.827+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbed00960b0 con 0x7fbed40ffb90 2024-03-18T20:24:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:56 smithi169 ceph-mon[48529]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-18T20:24:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:56 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/88756930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:24:56.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:56 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1545753371' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:24:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.036+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbed4067180 con 0x7fbed40ffb90 2024-03-18T20:24:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.037+0000 7fbecaffd700 1 -- 172.21.15.98:0/1001287803 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7fbed00996d0 con 0x7fbed40ffb90 2024-03-18T20:24:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:24:57.042 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.039+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 msgr2=0x7fbec00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.039+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 0x7fbec00a3660 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbec400c010 tx=0x7fbec4003720 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.039+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 msgr2=0x7fbed4069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.039+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbed000d980 tx=0x7fbed000dc90 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 shutdown_connections 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbed4101720 0x7fbed41a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed40ffb90 0x7fbed4069b10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 --2- 172.21.15.98:0/1001287803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbed4100d80 0x7fbed419dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 >> 172.21.15.98:0/1001287803 conn(0x7fbed40fb100 msgr2=0x7fbed41041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 shutdown_connections 2024-03-18T20:24:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.040+0000 7fbedc0be700 1 -- 172.21.15.98:0/1001287803 wait complete. 2024-03-18T20:24:57.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/1225623230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100180 msgr2=0x7fb1d01005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/1225623230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100180 0x7fb1d01005a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009990 tx=0x7fb1c4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/1225623230 shutdown_connections 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/1225623230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0101850 0x7fb1d01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/1225623230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100180 0x7fb1d01005a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/1225623230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0100e90 0x7fb1d0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/1225623230 >> 172.21.15.98:0/1225623230 conn(0x7fb1d00fa950 msgr2=0x7fb1d00fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/1225623230 shutdown_connections 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/1225623230 wait complete. 2024-03-18T20:24:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.177+0000 7fb1d5bd6700 1 Processor -- start 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 -- start start 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0100180 0x7fb1d00ff810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 0x7fb1d019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d019e830 con 0x7fb1d0100e90 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d019e9a0 con 0x7fb1d0100180 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1d5bd6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d019eb60 con 0x7fb1d0101850 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60242/0 (socket says 172.21.15.98:60242) 2024-03-18T20:24:57.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 -- 172.21.15.98:0/2228688801 learned_addr learned my addr 172.21.15.98:0/2228688801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1cf7fe700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0100180 0x7fb1d00ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1cffff700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 0x7fb1d019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 -- 172.21.15.98:0/2228688801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 msgr2=0x7fb1d019e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 0x7fb1d019e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 -- 172.21.15.98:0/2228688801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0100180 msgr2=0x7fb1d00ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0100180 0x7fb1d00ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 -- 172.21.15.98:0/2228688801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40095f0 con 0x7fb1d0100e90 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1cffff700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 0x7fb1d019e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.178+0000 7fb1ceffd700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000b500 tx=0x7fb1c000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.179+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0011830 con 0x7fb1d0100e90 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.179+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb1c0011e20 con 0x7fb1d0100e90 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.179+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d019ede0 con 0x7fb1d0100e90 2024-03-18T20:24:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.179+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0010dd0 con 0x7fb1d0100e90 2024-03-18T20:24:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.179+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d019f200 con 0x7fb1d0100e90 2024-03-18T20:24:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.180+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d01071e0 con 0x7fb1d0100e90 2024-03-18T20:24:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.181+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb1c0011990 con 0x7fb1d0100e90 2024-03-18T20:24:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.181+0000 7fb1ccff9700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 0x7fb1b80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.181+0000 7fb1cf7fe700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 0x7fb1b80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.181+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb1c00cc580 con 0x7fb1d0100e90 2024-03-18T20:24:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.182+0000 7fb1cf7fe700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 0x7fb1b80a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b5c0 tx=0x7fb1c40036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:57.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.185+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1c0095b20 con 0x7fb1d0100e90 2024-03-18T20:24:57.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.434+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb1d0067260 con 0x7fb1d0100e90 2024-03-18T20:24:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.449+0000 7fb1ccff9700 1 -- 172.21.15.98:0/2228688801 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb1c0099140 con 0x7fb1d0100e90 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 msgr2=0x7fb1b80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 0x7fb1b80a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b5c0 tx=0x7fb1c40036f0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 msgr2=0x7fb1d019ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000b500 tx=0x7fb1c000b810 comp rx=0 tx=0).stop 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 shutdown_connections 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1d0101850 0x7fb1d019e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb1b80a1120 0x7fb1b80a35e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1d0100180 0x7fb1d00ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 --2- 172.21.15.98:0/2228688801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1d0100e90 0x7fb1d019ba20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 >> 172.21.15.98:0/2228688801 conn(0x7fb1d00fa950 msgr2=0x7fb1d0104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 shutdown_connections 2024-03-18T20:24:57.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.451+0000 7fb1d5bd6700 1 -- 172.21.15.98:0/2228688801 wait complete. 2024-03-18T20:24:57.468 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-18T20:24:57.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-18T20:24:57.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 -- 172.21.15.98:0/1705949780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 msgr2=0x7f01c01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 --2- 172.21.15.98:0/1705949780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c01011a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f01b00099c0 tx=0x7f01b0009cd0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 -- 172.21.15.98:0/1705949780 shutdown_connections 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 --2- 172.21.15.98:0/1705949780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0102910 0x7f01c01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 --2- 172.21.15.98:0/1705949780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c01011a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 --2- 172.21.15.98:0/1705949780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0101f70 0x7f01c01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 -- 172.21.15.98:0/1705949780 >> 172.21.15.98:0/1705949780 conn(0x7f01c00fc2f0 msgr2=0x7f01c00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 -- 172.21.15.98:0/1705949780 shutdown_connections 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.589+0000 7f01c59db700 1 -- 172.21.15.98:0/1705949780 wait complete. 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 Processor -- start 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 -- start start 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0101f70 0x7f01c00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0102910 0x7f01c00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0073e80 con 0x7f01c0100d80 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0073ff0 con 0x7f01c0102910 2024-03-18T20:24:57.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01c59db700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01a0c00 con 0x7f01c0101f70 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.590+0000 7f01beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60260/0 (socket says 172.21.15.98:60260) 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 -- 172.21.15.98:0/735678216 learned_addr learned my addr 172.21.15.98:0/735678216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01bf7fe700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0102910 0x7f01c00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01be7fc700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0101f70 0x7f01c00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 -- 172.21.15.98:0/735678216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0101f70 msgr2=0x7f01c00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0101f70 0x7f01c00731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 -- 172.21.15.98:0/735678216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0102910 msgr2=0x7f01c00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0102910 0x7f01c00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 -- 172.21.15.98:0/735678216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00095f0 con 0x7f01c0100d80 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01beffd700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009f50 tx=0x7f01b0005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b001d070 con 0x7f01c0100d80 2024-03-18T20:24:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f01b000bda0 con 0x7f01c0100d80 2024-03-18T20:24:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c01a0e30 con 0x7f01c0100d80 2024-03-18T20:24:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0022d20 con 0x7f01c0100d80 2024-03-18T20:24:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.591+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c01a11f0 con 0x7f01c0100d80 2024-03-18T20:24:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.592+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c0108250 con 0x7f01c0100d80 2024-03-18T20:24:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.593+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f01b0003f80 con 0x7f01c0100d80 2024-03-18T20:24:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.594+0000 7f01c49d9700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 0x7f01a80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.594+0000 7f01be7fc700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 0x7f01a80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.594+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f01b00ce210 con 0x7f01c0100d80 2024-03-18T20:24:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.594+0000 7f01be7fc700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 0x7f01a80a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01b400ba60 tx=0x7f01b4005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.597+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f01b00977e0 con 0x7f01c0100d80 2024-03-18T20:24:57.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1713242129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1001287803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T20:24:57.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1713242129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1001287803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T20:24:57.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:57.840+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f01c002ccb0 con 0x7f01c0100d80 2024-03-18T20:24:57.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1713242129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1001287803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:24:57.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T20:24:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.457+0000 7f01c49d9700 1 -- 172.21.15.98:0/735678216 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f01b009ae00 con 0x7f01c0100d80 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 msgr2=0x7f01a80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 0x7f01a80a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01b400ba60 tx=0x7f01b4005d50 comp rx=0 tx=0).stop 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 msgr2=0x7f01c0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009f50 tx=0x7f01b0005b00 comp rx=0 tx=0).stop 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 shutdown_connections 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01a80a11a0 0x7f01a80a3660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01c0102910 0x7f01c00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c0100d80 0x7f01c0074b10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 --2- 172.21.15.98:0/735678216 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01c0101f70 0x7f01c00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 >> 172.21.15.98:0/735678216 conn(0x7f01c00fc2f0 msgr2=0x7f01c0105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 shutdown_connections 2024-03-18T20:24:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.460+0000 7f01c59db700 1 -- 172.21.15.98:0/735678216 wait complete. 2024-03-18T20:24:58.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 -- 172.21.15.98:0/2299680333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 msgr2=0x7f73b4101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 --2- 172.21.15.98:0/2299680333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b4101610 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009990 tx=0x7f73a8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 -- 172.21.15.98:0/2299680333 shutdown_connections 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 --2- 172.21.15.98:0/2299680333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b41008c0 0x7f73b4108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 --2- 172.21.15.98:0/2299680333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b4101610 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 --2- 172.21.15.98:0/2299680333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b40fff20 0x7f73b4100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 -- 172.21.15.98:0/2299680333 >> 172.21.15.98:0/2299680333 conn(0x7f73b40fa950 msgr2=0x7f73b40fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 -- 172.21.15.98:0/2299680333 shutdown_connections 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.603+0000 7f73bb242700 1 -- 172.21.15.98:0/2299680333 wait complete. 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 Processor -- start 2024-03-18T20:24:58.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 -- start start 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b40fff20 0x7f73b40ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b41008c0 0x7f73b419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b419e6e0 con 0x7f73b41011f0 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b419e850 con 0x7f73b40fff20 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73bb242700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b419ea10 con 0x7f73b41008c0 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60274/0 (socket says 172.21.15.98:60274) 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 -- 172.21.15.98:0/2607338736 learned_addr learned my addr 172.21.15.98:0/2607338736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b8fde700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b40fff20 0x7f73b40ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b3fff700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b41008c0 0x7f73b419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 -- 172.21.15.98:0/2607338736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b41008c0 msgr2=0x7f73b419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:58.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b41008c0 0x7f73b419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 -- 172.21.15.98:0/2607338736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b40fff20 msgr2=0x7f73b40ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b40fff20 0x7f73b40ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 -- 172.21.15.98:0/2607338736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a80095f0 con 0x7f73b41011f0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.604+0000 7f73b97df700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f73ac007ee0 tx=0x7f73ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.605+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac017070 con 0x7f73b41011f0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.605+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f73ac00f040 con 0x7f73b41011f0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.605+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b419ecf0 con 0x7f73b41011f0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.605+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac005710 con 0x7f73b41011f0 2024-03-18T20:24:58.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.605+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b419f1b0 con 0x7f73b41011f0 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.606+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b4107260 con 0x7f73b41011f0 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.607+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f73ac005870 con 0x7f73b41011f0 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.607+0000 7f73b1ffb700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 0x7f73a40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.607+0000 7f73b8fde700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 0x7f73a40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.607+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f73ac0cc2c0 con 0x7f73b41011f0 2024-03-18T20:24:58.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.608+0000 7f73b8fde700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 0x7f73a40a35e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f73a8006010 tx=0x7f73a800b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:58.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.612+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73ac095860 con 0x7f73b41011f0 2024-03-18T20:24:58.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T20:24:58.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:58 smithi136 ceph-mon[48073]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:58.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:58 smithi136 ceph-mon[48073]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-03-18T20:24:58.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T20:24:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T20:24:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:58 smithi098 ceph-mon[43462]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:58 smithi098 ceph-mon[43462]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-03-18T20:24:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T20:24:58.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:58.854+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f73b402d130 con 0x7f73b41011f0 2024-03-18T20:24:58.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2228688801' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T20:24:58.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:58 smithi169 ceph-mon[48529]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:58 smithi169 ceph-mon[48529]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-03-18T20:24:58.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T20:24:59.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.462+0000 7f73b1ffb700 1 -- 172.21.15.98:0/2607338736 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f73b402d130 con 0x7f73b41011f0 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 msgr2=0x7f73a40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 0x7f73a40a35e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f73a8006010 tx=0x7f73a800b610 comp rx=0 tx=0).stop 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 msgr2=0x7f73b419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f73ac007ee0 tx=0x7f73ac00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 shutdown_connections 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b41011f0 0x7f73b419df40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73a40a1120 0x7f73a40a35e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73b40fff20 0x7f73b40ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 --2- 172.21.15.98:0/2607338736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73b41008c0 0x7f73b419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 >> 172.21.15.98:0/2607338736 conn(0x7f73b40fa950 msgr2=0x7f73b41043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 shutdown_connections 2024-03-18T20:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.465+0000 7f73bb242700 1 -- 172.21.15.98:0/2607338736 wait complete. 2024-03-18T20:24:59.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T20:24:59.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 -- 172.21.15.98:0/1213385910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0101850 msgr2=0x7f86e0103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:59.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 --2- 172.21.15.98:0/1213385910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0101850 0x7f86e0103c40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009990 tx=0x7f86d4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 -- 172.21.15.98:0/1213385910 shutdown_connections 2024-03-18T20:24:59.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 --2- 172.21.15.98:0/1213385910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0106ab0 0x7f86e0108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 --2- 172.21.15.98:0/1213385910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0101850 0x7f86e0103c40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 --2- 172.21.15.98:0/1213385910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 0x7f86e0106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 -- 172.21.15.98:0/1213385910 >> 172.21.15.98:0/1213385910 conn(0x7f86e00fb1c0 msgr2=0x7f86e00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 -- 172.21.15.98:0/1213385910 shutdown_connections 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.611+0000 7f86e7546700 1 -- 172.21.15.98:0/1213385910 wait complete. 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 Processor -- start 2024-03-18T20:24:59.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 -- start start 2024-03-18T20:24:59.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 0x7f86e0196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:59.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 0x7f86e0197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:59.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:59.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e019e700 con 0x7f86e0106ab0 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e019e840 con 0x7f86e0104180 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e7546700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e019e9d0 con 0x7f86e0101850 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e5ae3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e5ae3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60284/0 (socket says 172.21.15.98:60284) 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e5ae3700 1 -- 172.21.15.98:0/3708295533 learned_addr learned my addr 172.21.15.98:0/3708295533 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e4ae1700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 0x7f86e0197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.612+0000 7f86e52e2700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 0x7f86e0196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 -- 172.21.15.98:0/3708295533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 msgr2=0x7f86e0196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 0x7f86e0196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 -- 172.21.15.98:0/3708295533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 msgr2=0x7f86e0197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 0x7f86e0197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 -- 172.21.15.98:0/3708295533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40095f0 con 0x7f86e0106ab0 2024-03-18T20:24:59.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e52e2700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 0x7f86e0196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e5ae3700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f86d8007ee0 tx=0x7f86d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8017070 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e019ecb0 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f86d800f040 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8005710 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.613+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e019f170 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.614+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e00fcd40 con 0x7f86e0106ab0 2024-03-18T20:24:59.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.615+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f86d8005870 con 0x7f86e0106ab0 2024-03-18T20:24:59.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.615+0000 7f86d27fc700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 0x7f86cc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:24:59.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.616+0000 7f86e52e2700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 0x7f86cc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:24:59.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.616+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f86d80cc970 con 0x7f86e0106ab0 2024-03-18T20:24:59.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.616+0000 7f86e52e2700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 0x7f86cc0a35e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f86d40095c0 tx=0x7f86d4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:24:59.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.619+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f86d8095eb0 con 0x7f86e0106ab0 2024-03-18T20:24:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T20:24:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T20:24:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T20:24:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:59.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T20:24:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T20:24:59.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:24:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T20:24:59.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T20:24:59.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:24:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T20:24:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:24:59.860+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f86e0067f40 con 0x7f86e0106ab0 2024-03-18T20:24:59.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/735678216' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T20:24:59.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:24:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T20:25:00.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.476+0000 7f86d27fc700 1 -- 172.21.15.98:0/3708295533 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f86d80994d0 con 0x7f86e0106ab0 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 msgr2=0x7f86cc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 0x7f86cc0a35e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f86d40095c0 tx=0x7f86d4009510 comp rx=0 tx=0).stop 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 msgr2=0x7f86e019e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f86d8007ee0 tx=0x7f86d800d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 shutdown_connections 2024-03-18T20:25:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86e0106ab0 0x7f86e019e130 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f86e0101850 0x7f86e0196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f86cc0a1120 0x7f86cc0a35e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 --2- 172.21.15.98:0/3708295533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e0104180 0x7f86e0197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 >> 172.21.15.98:0/3708295533 conn(0x7f86e00fb1c0 msgr2=0x7f86e0102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 shutdown_connections 2024-03-18T20:25:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.479+0000 7f86e7546700 1 -- 172.21.15.98:0/3708295533 wait complete. 2024-03-18T20:25:00.500 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-18T20:25:00.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 -- 172.21.15.98:0/3909077482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154104180 msgr2=0x7f1154106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 --2- 172.21.15.98:0/3909077482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154104180 0x7f1154106570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1148009990 tx=0x7f1148009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 -- 172.21.15.98:0/3909077482 shutdown_connections 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 --2- 172.21.15.98:0/3909077482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154106ab0 0x7f1154108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 --2- 172.21.15.98:0/3909077482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 --2- 172.21.15.98:0/3909077482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154104180 0x7f1154106570 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 -- 172.21.15.98:0/3909077482 >> 172.21.15.98:0/3909077482 conn(0x7f11540fb200 msgr2=0x7f11540fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 -- 172.21.15.98:0/3909077482 shutdown_connections 2024-03-18T20:25:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.627+0000 7f1158ade700 1 -- 172.21.15.98:0/3909077482 wait complete. 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 Processor -- start 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 -- start start 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 0x7f11541971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419e780 con 0x7f1154106ab0 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419e8f0 con 0x7f1154101850 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1158ade700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419eab0 con 0x7f1154104180 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1152d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1152d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60294/0 (socket says 172.21.15.98:60294) 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.628+0000 7f1152d9d700 1 -- 172.21.15.98:0/391430542 learned_addr learned my addr 172.21.15.98:0/391430542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:00.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f115259c700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1151d9b700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 0x7f11541971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 -- 172.21.15.98:0/391430542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 msgr2=0x7f11541971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 0x7f11541971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 -- 172.21.15.98:0/391430542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 msgr2=0x7f1154196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 -- 172.21.15.98:0/391430542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480095f0 con 0x7f1154106ab0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f115259c700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1151d9b700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 0x7f11541971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1152d9d700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f114c007ee0 tx=0x7f114c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c017070 con 0x7f1154106ab0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f114c00f040 con 0x7f1154106ab0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115419ed90 con 0x7f1154106ab0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c005710 con 0x7f1154106ab0 2024-03-18T20:25:00.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.629+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115419f1e0 con 0x7f1154106ab0 2024-03-18T20:25:00.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.630+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11540fcd80 con 0x7f1154106ab0 2024-03-18T20:25:00.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.631+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f114c005870 con 0x7f1154106ab0 2024-03-18T20:25:00.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.631+0000 7f11437fe700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 0x7f113c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:00.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.631+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f114c0cc960 con 0x7f1154106ab0 2024-03-18T20:25:00.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.632+0000 7f115259c700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 0x7f113c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:00.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.632+0000 7f115259c700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 0x7f113c0a3660 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f1144005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:00.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.635+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f114c095f00 con 0x7f1154106ab0 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:active 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: Cluster is now healthy 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T20:25:00.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-18T20:25:00.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T20:25:00.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:active 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: Cluster is now healthy 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-18T20:25:00.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T20:25:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.862+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f115402d160 con 0x7f113c0a11a0 2024-03-18T20:25:00.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] up:active 2024-03-18T20:25:00.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2607338736' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T20:25:00.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T20:25:00.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: Cluster is now healthy 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-18T20:25:00.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T20:25:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.886+0000 7f11437fe700 1 -- 172.21.15.98:0/391430542 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f115402d160 con 0x7f113c0a11a0 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 msgr2=0x7f113c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 0x7f113c0a3660 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1144005fd0 tx=0x7f1144005de0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 msgr2=0x7f115419e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f114c007ee0 tx=0x7f114c00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 shutdown_connections 2024-03-18T20:25:00.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1154106ab0 0x7f115419e1b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f113c0a11a0 0x7f113c0a3660 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1154101850 0x7f1154196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 --2- 172.21.15.98:0/391430542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1154104180 0x7f11541971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 >> 172.21.15.98:0/391430542 conn(0x7f11540fb200 msgr2=0x7f1154103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 shutdown_connections 2024-03-18T20:25:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:00.888+0000 7f1158ade700 1 -- 172.21.15.98:0/391430542 wait complete. 2024-03-18T20:25:00.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T20:25:01.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.029+0000 7fe3b05b7700 1 -- 172.21.15.98:0/1126264405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a8100e60 msgr2=0x7fe3a81012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.029+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/1126264405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a8100e60 0x7fe3a81012c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe3980099c0 tx=0x7fe398009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- 172.21.15.98:0/1126264405 shutdown_connections 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/1126264405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 0x7fe3a81080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/1126264405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/1126264405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a8100e60 0x7fe3a81012c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- 172.21.15.98:0/1126264405 >> 172.21.15.98:0/1126264405 conn(0x7fe3a80fb1c0 msgr2=0x7fe3a80fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- 172.21.15.98:0/1126264405 shutdown_connections 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- 172.21.15.98:0/1126264405 wait complete. 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 Processor -- start 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- start start 2024-03-18T20:25:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 0x7fe3a8199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 0x7fe3a81a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a81a0a00 con 0x7fe3a80ffc70 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a81a0b40 con 0x7fe3a8100e60 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3b05b7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a81a0cd0 con 0x7fe3a8101800 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.030+0000 7fe3ae353700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60322/0 (socket says 172.21.15.98:60322) 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 -- 172.21.15.98:0/3696971313 learned_addr learned my addr 172.21.15.98:0/3696971313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3aeb54700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 0x7fe3a81a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 -- 172.21.15.98:0/3696971313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 msgr2=0x7fe3a81a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3adb52700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 0x7fe3a8199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 0x7fe3a81a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 -- 172.21.15.98:0/3696971313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 msgr2=0x7fe3a8199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 0x7fe3a8199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 -- 172.21.15.98:0/3696971313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980095f0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3adb52700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 0x7fe3a8199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3ae353700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400d980 tx=0x7fe3a400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a400de60 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe3a40044b0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a81a0fb0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4003b40 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.031+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a81a13d0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.032+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a8067f40 con 0x7fe3a80ffc70 2024-03-18T20:25:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.033+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe3a4003ca0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.034+0000 7fe39f7fe700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 0x7fe3940a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.034+0000 7fe3adb52700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 0x7fe3940a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.034+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3a40cc7a0 con 0x7fe3a80ffc70 2024-03-18T20:25:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.034+0000 7fe3adb52700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 0x7fe3940a35e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe398003780 tx=0x7fe398000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.038+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3a4095d40 con 0x7fe3a80ffc70 2024-03-18T20:25:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.267+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3a81a16b0 con 0x7fe3940a1120 2024-03-18T20:25:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.290+0000 7fe39f7fe700 1 -- 172.21.15.98:0/3696971313 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe3a81a16b0 con 0x7fe3940a1120 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 msgr2=0x7fe3940a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 0x7fe3940a35e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe398003780 tx=0x7fe398000bc0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 msgr2=0x7fe3a8198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400d980 tx=0x7fe3a400dd40 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 shutdown_connections 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe3a8101800 0x7fe3a81a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3a80ffc70 0x7fe3a8198ed0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe3940a1120 0x7fe3940a35e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 --2- 172.21.15.98:0/3696971313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3a8100e60 0x7fe3a8199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 >> 172.21.15.98:0/3696971313 conn(0x7fe3a80fb1c0 msgr2=0x7fe3a8069380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 shutdown_connections 2024-03-18T20:25:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.292+0000 7fe3b05b7700 1 -- 172.21.15.98:0/3696971313 wait complete. 2024-03-18T20:25:01.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:25:00.867+0000 7f4a80d0d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:01.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:25:00.867+0000 7f4a80d0d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:01.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:25:00.867+0000 7f4a80d0d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:01.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:25:00.867+0000 7f4a80d0d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:01.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:25:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:25:00.867+0000 7f4a80d0d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:01.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T20:25:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 -- 172.21.15.98:0/1465033531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8101670 msgr2=0x7fb7c8107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/1465033531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8101670 0x7fb7c8107f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000b0d0 tx=0x7fb7c000b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 -- 172.21.15.98:0/1465033531 shutdown_connections 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/1465033531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c8101670 0x7fb7c8107f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/1465033531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c80ffae0 0x7fb7c80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/1465033531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 0x7fb7c8101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 -- 172.21.15.98:0/1465033531 >> 172.21.15.98:0/1465033531 conn(0x7fb7c80fb070 msgr2=0x7fb7c80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 -- 172.21.15.98:0/1465033531 shutdown_connections 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 -- 172.21.15.98:0/1465033531 wait complete. 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.433+0000 7fb7ccda1700 1 Processor -- start 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 -- start start 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 0x7fb7c819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c8101670 0x7fb7c819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819e620 con 0x7fb7c80ffae0 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819e790 con 0x7fb7c8100cd0 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7ccda1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819e950 con 0x7fb7c8101670 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60330/0 (socket says 172.21.15.98:60330) 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 -- 172.21.15.98:0/3616685167 learned_addr learned my addr 172.21.15.98:0/3616685167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c6ffd700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 0x7fb7c819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c7fff700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c8101670 0x7fb7c819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 -- 172.21.15.98:0/3616685167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c8101670 msgr2=0x7fb7c819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c8101670 0x7fb7c819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 -- 172.21.15.98:0/3616685167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 msgr2=0x7fb7c819b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 0x7fb7c819b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 -- 172.21.15.98:0/3616685167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0009d00 con 0x7fb7c80ffae0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c77fe700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d980 tx=0x7fb7bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.434+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc004020 con 0x7fb7c80ffae0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.435+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c819ec30 con 0x7fb7c80ffae0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.435+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb7bc00f690 con 0x7fb7c80ffae0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.435+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0039e0 con 0x7fb7c80ffae0 2024-03-18T20:25:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.435+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c819f140 con 0x7fb7c80ffae0 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.436+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8106f10 con 0x7fb7c80ffae0 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.437+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb7bc00f800 con 0x7fb7c80ffae0 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.437+0000 7fb7c4ff9700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 0x7fb7b00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.437+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb7bc0cc6b0 con 0x7fb7c80ffae0 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.437+0000 7fb7c6ffd700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 0x7fb7b00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.438+0000 7fb7c6ffd700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 0x7fb7b00a35e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8069ec0 tx=0x7fb7b80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.442+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7bc095c50 con 0x7fb7c80ffae0 2024-03-18T20:25:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.489+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb7bc096020 con 0x7fb7c80ffae0 2024-03-18T20:25:01.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:01 smithi098 ceph-mon[43462]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:active 2024-03-18T20:25:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T20:25:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:01 smithi098 ceph-mon[43462]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-18T20:25:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:01 smithi098 ceph-mon[43462]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:25:01.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:01 smithi098 ceph-mon[43462]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:01.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.684+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7c80611f0 con 0x7fb7b00a1120 2024-03-18T20:25:01.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.689+0000 7fb7c4ff9700 1 -- 172.21.15.98:0/3616685167 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fb7c80611f0 con 0x7fb7b00a1120 2024-03-18T20:25:01.693 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/1cf87f69-e1b6-4c60-968f-b28ce3aef5d3 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 msgr2=0x7fb7b00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 0x7fb7b00a35e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8069ec0 tx=0x7fb7b80073d0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 msgr2=0x7fb7c8069960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d980 tx=0x7fb7bc00dc90 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 shutdown_connections 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7c8101670 0x7fb7c819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c80ffae0 0x7fb7c8069960 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb7b00a1120 0x7fb7b00a35e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 --2- 172.21.15.98:0/3616685167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7c8100cd0 0x7fb7c819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 >> 172.21.15.98:0/3616685167 conn(0x7fb7c80fb070 msgr2=0x7fb7c8104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 shutdown_connections 2024-03-18T20:25:01.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.691+0000 7fb7ccda1700 1 -- 172.21.15.98:0/3616685167 wait complete. 2024-03-18T20:25:01.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T20:25:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:01 smithi136 ceph-mon[48073]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:active 2024-03-18T20:25:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T20:25:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:01 smithi136 ceph-mon[48073]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-18T20:25:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:01 smithi136 ceph-mon[48073]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:25:01.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:01 smithi136 ceph-mon[48073]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- 172.21.15.98:0/620720767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd750101890 msgr2=0x7fd750103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- 172.21.15.98:0/620720767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd750101890 0x7fd750103c80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd744009990 tx=0x7fd744009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- 172.21.15.98:0/620720767 shutdown_connections 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- 172.21.15.98:0/620720767 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 0x7fd750108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- 172.21.15.98:0/620720767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd750101890 0x7fd750103c80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- 172.21.15.98:0/620720767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7501041c0 0x7fd7501065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- 172.21.15.98:0/620720767 >> 172.21.15.98:0/620720767 conn(0x7fd7500fb200 msgr2=0x7fd7500fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- 172.21.15.98:0/620720767 shutdown_connections 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- 172.21.15.98:0/620720767 wait complete. 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 Processor -- start 2024-03-18T20:25:01.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- start start 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd750101890 0x7fd7501011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 0x7fd7500ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd750100520 con 0x7fd7501041c0 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd750100690 con 0x7fd750101890 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.831+0000 7fd756d77700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501a0d60 con 0x7fd750106af0 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60338/0 (socket says 172.21.15.98:60338) 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 -- 172.21.15.98:0/4027320363 learned_addr learned my addr 172.21.15.98:0/4027320363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd755314700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 0x7fd7500ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 -- 172.21.15.98:0/4027320363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 msgr2=0x7fd7500ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 0x7fd7500ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 -- 172.21.15.98:0/4027320363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd750101890 msgr2=0x7fd7501011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd754b13700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd750101890 0x7fd7501011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd750101890 0x7fd7501011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 -- 172.21.15.98:0/4027320363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7440095f0 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74ffff700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00b530 tx=0x7fd73c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c011830 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd73c011e20 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7501a0ff0 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c010dd0 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.832+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7501a1470 con 0x7fd7501041c0 2024-03-18T20:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.833+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7500fcd80 con 0x7fd7501041c0 2024-03-18T20:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.834+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd73c011990 con 0x7fd7501041c0 2024-03-18T20:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.835+0000 7fd74dffb700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 0x7fd7400a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.835+0000 7fd754b13700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 0x7fd7400a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.835+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd73c0cc710 con 0x7fd7501041c0 2024-03-18T20:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.835+0000 7fd754b13700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 0x7fd7400a3670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd744003780 tx=0x7fd744000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:01.839+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd73c095c60 con 0x7fd7501041c0 2024-03-18T20:25:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:01 smithi169 ceph-mon[48529]: mds.? [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] up:active 2024-03-18T20:25:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3708295533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T20:25:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:01 smithi169 ceph-mon[48529]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-18T20:25:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:01 smithi169 ceph-mon[48529]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:25:01.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:01 smithi169 ceph-mon[48529]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.068+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7500611f0 con 0x7fd7400a11b0 2024-03-18T20:25:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.092+0000 7fd74dffb700 1 -- 172.21.15.98:0/4027320363 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd7500611f0 con 0x7fd7400a11b0 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 msgr2=0x7fd7400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 0x7fd7400a3670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd744003780 tx=0x7fd744000bc0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 msgr2=0x7fd7500ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00b530 tx=0x7fd73c00b8f0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 shutdown_connections 2024-03-18T20:25:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd750106af0 0x7fd7500ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd7400a11b0 0x7fd7400a3670 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd750101890 0x7fd7501011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 --2- 172.21.15.98:0/4027320363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7501041c0 0x7fd7500ff840 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.094+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 >> 172.21.15.98:0/4027320363 conn(0x7fd7500fb200 msgr2=0x7fd750103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.095+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 shutdown_connections 2024-03-18T20:25:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.095+0000 7fd756d77700 1 -- 172.21.15.98:0/4027320363 wait complete. 2024-03-18T20:25:02.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T20:25:02.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/1382610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f440ff3e0 msgr2=0x7f7f440ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/1382610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f440ff3e0 0x7f7f440ff800 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7f340099c0 tx=0x7f7f34009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/1382610648 shutdown_connections 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/1382610648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 0x7f7f44107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/1382610648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f440ff3e0 0x7f7f440ff800 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/1382610648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f441005d0 0x7f7f44100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/1382610648 >> 172.21.15.98:0/1382610648 conn(0x7f7f440fa950 msgr2=0x7f7f440fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/1382610648 shutdown_connections 2024-03-18T20:25:02.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/1382610648 wait complete. 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 Processor -- start 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.234+0000 7f7f4bc9d700 1 -- start start 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f440ff3e0 0x7f7f44194a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 0x7f7f4419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4419c560 con 0x7f7f441005d0 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4419c6a0 con 0x7f7f440ff3e0 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4419c830 con 0x7f7f44100f70 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60356/0 (socket says 172.21.15.98:60356) 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 -- 172.21.15.98:0/803796830 learned_addr learned my addr 172.21.15.98:0/803796830 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49a39700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f440ff3e0 0x7f7f44194a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4a23a700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 0x7f7f4419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 -- 172.21.15.98:0/803796830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 msgr2=0x7f7f4419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 0x7f7f4419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 -- 172.21.15.98:0/803796830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f440ff3e0 msgr2=0x7f7f44194a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f440ff3e0 0x7f7f44194a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 -- 172.21.15.98:0/803796830 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340095f0 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f49238700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000d950 tx=0x7f7f4000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40009940 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7f40010460 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000f620 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4419cb10 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.235+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4419d020 con 0x7f7f441005d0 2024-03-18T20:25:02.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.236+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f44106900 con 0x7f7f441005d0 2024-03-18T20:25:02.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.237+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f7f400105d0 con 0x7f7f441005d0 2024-03-18T20:25:02.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.238+0000 7f7f3affd700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 0x7f7f300a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.238+0000 7f7f49a39700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 0x7f7f300a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.238+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7f400cca20 con 0x7f7f441005d0 2024-03-18T20:25:02.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.239+0000 7f7f49a39700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 0x7f7f300a36f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7f34006010 tx=0x7f7f34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.242+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f40095e60 con 0x7f7f441005d0 2024-03-18T20:25:02.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.478+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f440611f0 con 0x7f7f300a1230 2024-03-18T20:25:02.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.482+0000 7f7f3affd700 1 -- 172.21.15.98:0/803796830 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f7f440611f0 con 0x7f7f300a1230 2024-03-18T20:25:02.486 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/3a0e9182-74ff-4c2e-92c5-dc72e04e1ab4 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 msgr2=0x7f7f300a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 0x7f7f300a36f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7f34006010 tx=0x7f7f34000bc0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 msgr2=0x7f7f44194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000d950 tx=0x7f7f4000dc60 comp rx=0 tx=0).stop 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 shutdown_connections 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f44100f70 0x7f7f4419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7f300a1230 0x7f7f300a36f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f440ff3e0 0x7f7f44194a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 --2- 172.21.15.98:0/803796830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441005d0 0x7f7f44194f70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 >> 172.21.15.98:0/803796830 conn(0x7f7f440fa950 msgr2=0x7f7f44103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 shutdown_connections 2024-03-18T20:25:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.484+0000 7f7f4bc9d700 1 -- 172.21.15.98:0/803796830 wait complete. 2024-03-18T20:25:02.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:02 smithi098 ceph-mon[43462]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:02 smithi098 ceph-mon[43462]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-18T20:25:02.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:02 smithi098 ceph-mon[43462]: pgmap v233: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-18T20:25:02.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:02 smithi098 ceph-mon[43462]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- 172.21.15.98:0/701680661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 msgr2=0x7fe7fc107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- 172.21.15.98:0/701680661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc107fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400b0d0 tx=0x7fe7f400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- 172.21.15.98:0/701680661 shutdown_connections 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- 172.21.15.98:0/701680661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc107fd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- 172.21.15.98:0/701680661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc0fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- 172.21.15.98:0/701680661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc100d80 0x7fe7fc1011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- 172.21.15.98:0/701680661 >> 172.21.15.98:0/701680661 conn(0x7fe7fc0fb100 msgr2=0x7fe7fc0fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- 172.21.15.98:0/701680661 shutdown_connections 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- 172.21.15.98:0/701680661 wait complete. 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 Processor -- start 2024-03-18T20:25:02.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- start start 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc100d80 0x7fe7fc195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc19d590 con 0x7fe7fc101720 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc19d6d0 con 0x7fe7fc100d80 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.619+0000 7fe8012aa700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc19d860 con 0x7fe7fc0ffb90 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60372/0 (socket says 172.21.15.98:60372) 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 -- 172.21.15.98:0/3053116688 learned_addr learned my addr 172.21.15.98:0/3053116688 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7f3fff700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc100d80 0x7fe7fc195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7faffd700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 -- 172.21.15.98:0/3053116688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 msgr2=0x7fe7fc195a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc195a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 -- 172.21.15.98:0/3053116688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc100d80 msgr2=0x7fe7fc195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc100d80 0x7fe7fc195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 -- 172.21.15.98:0/3053116688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4009d00 con 0x7fe7fc101720 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7faffd700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7fb7fe700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4007af0 tx=0x7fe7f4007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400e040 con 0x7fe7fc101720 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7f401db00 con 0x7fe7fc101720 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc19dae0 con 0x7fe7fc101720 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f401b990 con 0x7fe7fc101720 2024-03-18T20:25:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.620+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc19dfb0 con 0x7fe7fc101720 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.622+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc106fb0 con 0x7fe7fc101720 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.622+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe7f401dc70 con 0x7fe7fc101720 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.623+0000 7fe7f8ff9700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 0x7fe7e80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.623+0000 7fe7faffd700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 0x7fe7e80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.623+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7f40ce180 con 0x7fe7fc101720 2024-03-18T20:25:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.623+0000 7fe7faffd700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 0x7fe7e80a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc100be0 tx=0x7fe7ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.627+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe7f4097650 con 0x7fe7fc101720 2024-03-18T20:25:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:02 smithi136 ceph-mon[48073]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:02 smithi136 ceph-mon[48073]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-18T20:25:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:02 smithi136 ceph-mon[48073]: pgmap v233: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-18T20:25:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:02 smithi136 ceph-mon[48073]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.867+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7fc02d140 con 0x7fe7fc101720 2024-03-18T20:25:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.868+0000 7fe7f8ff9700 1 -- 172.21.15.98:0/3053116688 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7fe7f409ac70 con 0x7fe7fc101720 2024-03-18T20:25:02.873 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:02.873 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 msgr2=0x7fe7e80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 0x7fe7e80a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc100be0 tx=0x7fe7ec009260 comp rx=0 tx=0).stop 2024-03-18T20:25:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 msgr2=0x7fe7fc19cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4007af0 tx=0x7fe7f4007850 comp rx=0 tx=0).stop 2024-03-18T20:25:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 shutdown_connections 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7fc101720 0x7fe7fc19cfc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7fc0ffb90 0x7fe7fc195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe7e80a1230 0x7fe7e80a36f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 --2- 172.21.15.98:0/3053116688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc100d80 0x7fe7fc195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 >> 172.21.15.98:0/3053116688 conn(0x7fe7fc0fb100 msgr2=0x7fe7fc1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 shutdown_connections 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:02.870+0000 7fe8012aa700 1 -- 172.21.15.98:0/3053116688 wait complete. 2024-03-18T20:25:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:02 smithi169 ceph-mon[48529]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:02 smithi169 ceph-mon[48529]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-18T20:25:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:02 smithi169 ceph-mon[48529]: pgmap v233: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-18T20:25:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:02 smithi169 ceph-mon[48529]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:02.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:03.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 -- 172.21.15.98:0/3911504664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 msgr2=0x7f389c109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3911504664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c109180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f389400b0a0 tx=0x7f389400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 -- 172.21.15.98:0/3911504664 shutdown_connections 2024-03-18T20:25:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3911504664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c109180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3911504664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c0fef60 0x7f389c0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3911504664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c101f30 0x7f389c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.004+0000 7f38a42cb700 1 -- 172.21.15.98:0/3911504664 >> 172.21.15.98:0/3911504664 conn(0x7f389c0fa890 msgr2=0x7f389c0fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- 172.21.15.98:0/3911504664 shutdown_connections 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- 172.21.15.98:0/3911504664 wait complete. 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 Processor -- start 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- start start 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c0fef60 0x7f389c196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c101f30 0x7f389c197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c19e730 con 0x7f389c1028d0 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c19e870 con 0x7f389c101f30 2024-03-18T20:25:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a42cb700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c19ea00 con 0x7f389c0fef60 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a2868700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a2868700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60402/0 (socket says 172.21.15.98:60402) 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a2868700 1 -- 172.21.15.98:0/3514191263 learned_addr learned my addr 172.21.15.98:0/3514191263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.005+0000 7f38a1866700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c101f30 0x7f389c197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2067700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c0fef60 0x7f389c196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 -- 172.21.15.98:0/3514191263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c0fef60 msgr2=0x7f389c196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c0fef60 0x7f389c196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 -- 172.21.15.98:0/3514191263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c101f30 msgr2=0x7f389c197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c101f30 0x7f389c197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 -- 172.21.15.98:0/3514191263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894009d00 con 0x7f389c1028d0 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a2868700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3894004d00 tx=0x7f3894004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389400e040 con 0x7f389c1028d0 2024-03-18T20:25:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f389400cec0 con 0x7f389c1028d0 2024-03-18T20:25:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c19ec80 con 0x7f389c1028d0 2024-03-18T20:25:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389401ba30 con 0x7f389c1028d0 2024-03-18T20:25:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.006+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c19f100 con 0x7f389c1028d0 2024-03-18T20:25:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.007+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c1081c0 con 0x7f389c1028d0 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.008+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f389401bb90 con 0x7f389c1028d0 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.008+0000 7f38937fe700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 0x7f38880a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.009+0000 7f38a2067700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 0x7f38880a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.009+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f38940cee70 con 0x7f389c1028d0 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.009+0000 7f38a2067700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 0x7f38880a3770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f389c069050 tx=0x7f389800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.012+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3894098400 con 0x7f389c1028d0 2024-03-18T20:25:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.216+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f389c067e60 con 0x7f389c1028d0 2024-03-18T20:25:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.218+0000 7f38937fe700 1 -- 172.21.15.98:0/3514191263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f3894021090 con 0x7f389c1028d0 2024-03-18T20:25:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:03.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 msgr2=0x7f38880a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 0x7f38880a3770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f389c069050 tx=0x7f389800b410 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 msgr2=0x7f389c19e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3894004d00 tx=0x7f3894004d30 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 shutdown_connections 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f389c1028d0 0x7f389c19e160 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f389c0fef60 0x7f389c196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f38880a12b0 0x7f38880a3770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 --2- 172.21.15.98:0/3514191263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f389c101f30 0x7f389c197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 >> 172.21.15.98:0/3514191263 conn(0x7f389c0fa890 msgr2=0x7f389c105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 shutdown_connections 2024-03-18T20:25:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.220+0000 7f38a42cb700 1 -- 172.21.15.98:0/3514191263 wait complete. 2024-03-18T20:25:03.237 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:25:03.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 -- 172.21.15.98:0/1432691209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5580101780 msgr2=0x7f5580103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 --2- 172.21.15.98:0/1432691209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5580101780 0x7f5580103b70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5570009990 tx=0x7f5570009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 -- 172.21.15.98:0/1432691209 shutdown_connections 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 --2- 172.21.15.98:0/1432691209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f5580108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 --2- 172.21.15.98:0/1432691209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5580101780 0x7f5580103b70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 --2- 172.21.15.98:0/1432691209 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55801040b0 0x7f55801064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 -- 172.21.15.98:0/1432691209 >> 172.21.15.98:0/1432691209 conn(0x7f55800fb0d0 msgr2=0x7f55800fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 -- 172.21.15.98:0/1432691209 shutdown_connections 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.354+0000 7f5585065700 1 -- 172.21.15.98:0/1432691209 wait complete. 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 Processor -- start 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 -- start start 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5580101780 0x7f5580196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f558019e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019e650 con 0x7f55801040b0 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019e790 con 0x7f5580101780 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.355+0000 7f5585065700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019e920 con 0x7f55801069e0 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60986/0 (socket says 172.21.15.98:60986) 2024-03-18T20:25:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 -- 172.21.15.98:0/677790340 learned_addr learned my addr 172.21.15.98:0/677790340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557ed9d700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5580101780 0x7f5580196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557f59e700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f558019e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 -- 172.21.15.98:0/677790340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 msgr2=0x7f558019e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f558019e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 -- 172.21.15.98:0/677790340 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5580101780 msgr2=0x7f5580196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5580101780 0x7f5580196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 -- 172.21.15.98:0/677790340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55700095f0 con 0x7f55801040b0 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557f59e700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f558019e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f557e59c700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f557400d980 tx=0x7f557400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574004220 con 0x7f55801040b0 2024-03-18T20:25:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558019ec00 con 0x7f55801040b0 2024-03-18T20:25:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5574007640 con 0x7f55801040b0 2024-03-18T20:25:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740055e0 con 0x7f55801040b0 2024-03-18T20:25:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.356+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558019f110 con 0x7f55801040b0 2024-03-18T20:25:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.357+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55800fcc50 con 0x7f55801040b0 2024-03-18T20:25:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.358+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5574004750 con 0x7f55801040b0 2024-03-18T20:25:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.359+0000 7f5567fff700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 0x7f55680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.359+0000 7f557ed9d700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 0x7f55680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.359+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55740ccb20 con 0x7f55801040b0 2024-03-18T20:25:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.359+0000 7f557ed9d700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 0x7f55680a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f5570005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.363+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f55740cee60 con 0x7f55801040b0 2024-03-18T20:25:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.558+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55800611f0 con 0x7f55680a1230 2024-03-18T20:25:03.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.571+0000 7f5567fff700 1 -- 172.21.15.98:0/677790340 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+417358 (secure 0 0 0) 0x7f55800611f0 con 0x7f55680a1230 2024-03-18T20:25:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 msgr2=0x7f55680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 0x7f55680a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f5570005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 msgr2=0x7f5580197060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f557400d980 tx=0x7f557400dc90 comp rx=0 tx=0).stop 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 shutdown_connections 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55801069e0 0x7f558019e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55680a1230 0x7f55680a36f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5580101780 0x7f5580196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 --2- 172.21.15.98:0/677790340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55801040b0 0x7f5580197060 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 >> 172.21.15.98:0/677790340 conn(0x7f55800fb0d0 msgr2=0x7f5580103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 shutdown_connections 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.575+0000 7f5585065700 1 -- 172.21.15.98:0/677790340 wait complete. 2024-03-18T20:25:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-18T20:25:03.592 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-03-18T20:25:01.575125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":136,"num_read_kb":121,"num_write":284,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337064,"kb_used_data":12840,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604760,"statfs":{"total":1151940427776,"available":1151595274240,"internally_reserved":0,"allocated":13148160,"data_stored":7279950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3701,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":59,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430988+0000","last_change":"2024-03-18T20:24:41.360546+0000","last_active":"2024-03-18T20:24:45.430988+0000","last_peered":"2024-03-18T20:24:45.430988+0000","last_clean":"2024-03-18T20:24:45.430988+0000","last_became_active":"2024-03-18T20:24:41.360221+0000","last_became_peered":"2024-03-18T20:24:41.360221+0000","last_unstale":"2024-03-18T20:24:45.430988+0000","last_undegraded":"2024-03-18T20:24:45.430988+0000","last_fullsized":"2024-03-18T20:24:45.430988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:23.329362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441941+0000","last_change":"2024-03-18T20:24:41.354655+0000","last_active":"2024-03-18T20:24:45.441941+0000","last_peered":"2024-03-18T20:24:45.441941+0000","last_clean":"2024-03-18T20:24:45.441941+0000","last_became_active":"2024-03-18T20:24:41.354340+0000","last_became_peered":"2024-03-18T20:24:41.354340+0000","last_unstale":"2024-03-18T20:24:45.441941+0000","last_undegraded":"2024-03-18T20:24:45.441941+0000","last_fullsized":"2024-03-18T20:24:45.441941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:18.785662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440207+0000","last_change":"2024-03-18T20:24:41.351670+0000","last_active":"2024-03-18T20:24:45.440207+0000","last_peered":"2024-03-18T20:24:45.440207+0000","last_clean":"2024-03-18T20:24:45.440207+0000","last_became_active":"2024-03-18T20:24:41.351379+0000","last_became_peered":"2024-03-18T20:24:41.351379+0000","last_unstale":"2024-03-18T20:24:45.440207+0000","last_undegraded":"2024-03-18T20:24:45.440207+0000","last_fullsized":"2024-03-18T20:24:45.440207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:50.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436769+0000","last_change":"2024-03-18T20:24:41.364821+0000","last_active":"2024-03-18T20:24:45.436769+0000","last_peered":"2024-03-18T20:24:45.436769+0000","last_clean":"2024-03-18T20:24:45.436769+0000","last_became_active":"2024-03-18T20:24:41.364496+0000","last_became_peered":"2024-03-18T20:24:41.364496+0000","last_unstale":"2024-03-18T20:24:45.436769+0000","last_undegraded":"2024-03-18T20:24:45.436769+0000","last_fullsized":"2024-03-18T20:24:45.436769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:53.506013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431016+0000","last_change":"2024-03-18T20:24:41.356223+0000","last_active":"2024-03-18T20:24:45.431016+0000","last_peered":"2024-03-18T20:24:45.431016+0000","last_clean":"2024-03-18T20:24:45.431016+0000","last_became_active":"2024-03-18T20:24:41.355831+0000","last_became_peered":"2024-03-18T20:24:41.355831+0000","last_unstale":"2024-03-18T20:24:45.431016+0000","last_undegraded":"2024-03-18T20:24:45.431016+0000","last_fullsized":"2024-03-18T20:24:45.431016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:07.593472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852430+0000","last_change":"2024-03-18T20:24:41.367208+0000","last_active":"2024-03-18T20:24:45.852430+0000","last_peered":"2024-03-18T20:24:45.852430+0000","last_clean":"2024-03-18T20:24:45.852430+0000","last_became_active":"2024-03-18T20:24:41.366885+0000","last_became_peered":"2024-03-18T20:24:41.366885+0000","last_unstale":"2024-03-18T20:24:45.852430+0000","last_undegraded":"2024-03-18T20:24:45.852430+0000","last_fullsized":"2024-03-18T20:24:45.852430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:34.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436116+0000","last_change":"2024-03-18T20:24:41.367935+0000","last_active":"2024-03-18T20:24:45.436116+0000","last_peered":"2024-03-18T20:24:45.436116+0000","last_clean":"2024-03-18T20:24:45.436116+0000","last_became_active":"2024-03-18T20:24:41.367615+0000","last_became_peered":"2024-03-18T20:24:41.367615+0000","last_unstale":"2024-03-18T20:24:45.436116+0000","last_undegraded":"2024-03-18T20:24:45.436116+0000","last_fullsized":"2024-03-18T20:24:45.436116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:41.242857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426070+0000","last_change":"2024-03-18T20:24:41.369045+0000","last_active":"2024-03-18T20:24:45.426070+0000","last_peered":"2024-03-18T20:24:45.426070+0000","last_clean":"2024-03-18T20:24:45.426070+0000","last_became_active":"2024-03-18T20:24:41.368669+0000","last_became_peered":"2024-03-18T20:24:41.368669+0000","last_unstale":"2024-03-18T20:24:45.426070+0000","last_undegraded":"2024-03-18T20:24:45.426070+0000","last_fullsized":"2024-03-18T20:24:45.426070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:09.538243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432055+0000","last_change":"2024-03-18T20:24:41.366520+0000","last_active":"2024-03-18T20:24:45.432055+0000","last_peered":"2024-03-18T20:24:45.432055+0000","last_clean":"2024-03-18T20:24:45.432055+0000","last_became_active":"2024-03-18T20:24:41.366155+0000","last_became_peered":"2024-03-18T20:24:41.366155+0000","last_unstale":"2024-03-18T20:24:45.432055+0000","last_undegraded":"2024-03-18T20:24:45.432055+0000","last_fullsized":"2024-03-18T20:24:45.432055+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:33.151394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852478+0000","last_change":"2024-03-18T20:24:41.370742+0000","last_active":"2024-03-18T20:24:45.852478+0000","last_peered":"2024-03-18T20:24:45.852478+0000","last_clean":"2024-03-18T20:24:45.852478+0000","last_became_active":"2024-03-18T20:24:41.370438+0000","last_became_peered":"2024-03-18T20:24:41.370438+0000","last_unstale":"2024-03-18T20:24:45.852478+0000","last_undegraded":"2024-03-18T20:24:45.852478+0000","last_fullsized":"2024-03-18T20:24:45.852478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:04.331608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436604+0000","last_change":"2024-03-18T20:24:41.366970+0000","last_active":"2024-03-18T20:24:45.436604+0000","last_peered":"2024-03-18T20:24:45.436604+0000","last_clean":"2024-03-18T20:24:45.436604+0000","last_became_active":"2024-03-18T20:24:41.366440+0000","last_became_peered":"2024-03-18T20:24:41.366440+0000","last_unstale":"2024-03-18T20:24:45.436604+0000","last_undegraded":"2024-03-18T20:24:45.436604+0000","last_fullsized":"2024-03-18T20:24:45.436604+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:10.050606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436670+0000","last_change":"2024-03-18T20:24:41.372375+0000","last_active":"2024-03-18T20:24:45.436670+0000","last_peered":"2024-03-18T20:24:45.436670+0000","last_clean":"2024-03-18T20:24:45.436670+0000","last_became_active":"2024-03-18T20:24:41.371028+0000","last_became_peered":"2024-03-18T20:24:41.371028+0000","last_unstale":"2024-03-18T20:24:45.436670+0000","last_undegraded":"2024-03-18T20:24:45.436670+0000","last_fullsized":"2024-03-18T20:24:45.436670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:39.906774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430787+0000","last_change":"2024-03-18T20:24:41.368252+0000","last_active":"2024-03-18T20:24:45.430787+0000","last_peered":"2024-03-18T20:24:45.430787+0000","last_clean":"2024-03-18T20:24:45.430787+0000","last_became_active":"2024-03-18T20:24:41.367911+0000","last_became_peered":"2024-03-18T20:24:41.367911+0000","last_unstale":"2024-03-18T20:24:45.430787+0000","last_undegraded":"2024-03-18T20:24:45.430787+0000","last_fullsized":"2024-03-18T20:24:45.430787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:36.250924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440015+0000","last_change":"2024-03-18T20:24:41.358690+0000","last_active":"2024-03-18T20:24:45.440015+0000","last_peered":"2024-03-18T20:24:45.440015+0000","last_clean":"2024-03-18T20:24:45.440015+0000","last_became_active":"2024-03-18T20:24:41.358254+0000","last_became_peered":"2024-03-18T20:24:41.358254+0000","last_unstale":"2024-03-18T20:24:45.440015+0000","last_undegraded":"2024-03-18T20:24:45.440015+0000","last_fullsized":"2024-03-18T20:24:45.440015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:51.466799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440570+0000","last_change":"2024-03-18T20:24:41.364337+0000","last_active":"2024-03-18T20:24:45.440570+0000","last_peered":"2024-03-18T20:24:45.440570+0000","last_clean":"2024-03-18T20:24:45.440570+0000","last_became_active":"2024-03-18T20:24:41.364026+0000","last_became_peered":"2024-03-18T20:24:41.364026+0000","last_unstale":"2024-03-18T20:24:45.440570+0000","last_undegraded":"2024-03-18T20:24:45.440570+0000","last_fullsized":"2024-03-18T20:24:45.440570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:34.336403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431248+0000","last_change":"2024-03-18T20:24:41.360989+0000","last_active":"2024-03-18T20:24:45.431248+0000","last_peered":"2024-03-18T20:24:45.431248+0000","last_clean":"2024-03-18T20:24:45.431248+0000","last_became_active":"2024-03-18T20:24:41.360097+0000","last_became_peered":"2024-03-18T20:24:41.360097+0000","last_unstale":"2024-03-18T20:24:45.431248+0000","last_undegraded":"2024-03-18T20:24:45.431248+0000","last_fullsized":"2024-03-18T20:24:45.431248+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:18.522013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430879+0000","last_change":"2024-03-18T20:24:41.361221+0000","last_active":"2024-03-18T20:24:45.430879+0000","last_peered":"2024-03-18T20:24:45.430879+0000","last_clean":"2024-03-18T20:24:45.430879+0000","last_became_active":"2024-03-18T20:24:41.360924+0000","last_became_peered":"2024-03-18T20:24:41.360924+0000","last_unstale":"2024-03-18T20:24:45.430879+0000","last_undegraded":"2024-03-18T20:24:45.430879+0000","last_fullsized":"2024-03-18T20:24:45.430879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:24.233761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430265+0000","last_change":"2024-03-18T20:24:41.358145+0000","last_active":"2024-03-18T20:24:45.430265+0000","last_peered":"2024-03-18T20:24:45.430265+0000","last_clean":"2024-03-18T20:24:45.430265+0000","last_became_active":"2024-03-18T20:24:41.357862+0000","last_became_peered":"2024-03-18T20:24:41.357862+0000","last_unstale":"2024-03-18T20:24:45.430265+0000","last_undegraded":"2024-03-18T20:24:45.430265+0000","last_fullsized":"2024-03-18T20:24:45.430265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:01.582221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427416+0000","last_change":"2024-03-18T20:24:41.359214+0000","last_active":"2024-03-18T20:24:45.427416+0000","last_peered":"2024-03-18T20:24:45.427416+0000","last_clean":"2024-03-18T20:24:45.427416+0000","last_became_active":"2024-03-18T20:24:41.357976+0000","last_became_peered":"2024-03-18T20:24:41.357976+0000","last_unstale":"2024-03-18T20:24:45.427416+0000","last_undegraded":"2024-03-18T20:24:45.427416+0000","last_fullsized":"2024-03-18T20:24:45.427416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:36.873383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854870+0000","last_change":"2024-03-18T20:24:45.440427+0000","last_active":"2024-03-18T20:24:45.854870+0000","last_peered":"2024-03-18T20:24:45.854870+0000","last_clean":"2024-03-18T20:24:45.854870+0000","last_became_active":"2024-03-18T20:24:45.440023+0000","last_became_peered":"2024-03-18T20:24:45.440023+0000","last_unstale":"2024-03-18T20:24:45.854870+0000","last_undegraded":"2024-03-18T20:24:45.854870+0000","last_fullsized":"2024-03-18T20:24:45.854870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:37.745854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479306+0000","last_change":"2024-03-18T20:24:40.945829+0000","last_active":"2024-03-18T20:24:59.479306+0000","last_peered":"2024-03-18T20:24:59.479306+0000","last_clean":"2024-03-18T20:24:59.479306+0000","last_became_active":"2024-03-18T20:24:40.945498+0000","last_became_peered":"2024-03-18T20:24:40.945498+0000","last_unstale":"2024-03-18T20:24:59.479306+0000","last_undegraded":"2024-03-18T20:24:59.479306+0000","last_fullsized":"2024-03-18T20:24:59.479306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:40.221678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426446+0000","last_change":"2024-03-18T20:24:41.368911+0000","last_active":"2024-03-18T20:24:45.426446+0000","last_peered":"2024-03-18T20:24:45.426446+0000","last_clean":"2024-03-18T20:24:45.426446+0000","last_became_active":"2024-03-18T20:24:41.368603+0000","last_became_peered":"2024-03-18T20:24:41.368603+0000","last_unstale":"2024-03-18T20:24:45.426446+0000","last_undegraded":"2024-03-18T20:24:45.426446+0000","last_fullsized":"2024-03-18T20:24:45.426446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:37.722935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961438+0000","last_change":"2024-03-18T20:24:45.439904+0000","last_active":"2024-03-18T20:24:45.961438+0000","last_peered":"2024-03-18T20:24:45.961438+0000","last_clean":"2024-03-18T20:24:45.961438+0000","last_became_active":"2024-03-18T20:24:45.439489+0000","last_became_peered":"2024-03-18T20:24:45.439489+0000","last_unstale":"2024-03-18T20:24:45.961438+0000","last_undegraded":"2024-03-18T20:24:45.961438+0000","last_fullsized":"2024-03-18T20:24:45.961438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:21.696148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962086+0000","last_change":"2024-03-18T20:24:40.942150+0000","last_active":"2024-03-18T20:24:45.962086+0000","last_peered":"2024-03-18T20:24:45.962086+0000","last_clean":"2024-03-18T20:24:45.962086+0000","last_became_active":"2024-03-18T20:24:40.941848+0000","last_became_peered":"2024-03-18T20:24:40.941848+0000","last_unstale":"2024-03-18T20:24:45.962086+0000","last_undegraded":"2024-03-18T20:24:45.962086+0000","last_fullsized":"2024-03-18T20:24:45.962086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:44:08.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442127+0000","last_change":"2024-03-18T20:24:41.359112+0000","last_active":"2024-03-18T20:24:45.442127+0000","last_peered":"2024-03-18T20:24:45.442127+0000","last_clean":"2024-03-18T20:24:45.442127+0000","last_became_active":"2024-03-18T20:24:41.358817+0000","last_became_peered":"2024-03-18T20:24:41.358817+0000","last_unstale":"2024-03-18T20:24:45.442127+0000","last_undegraded":"2024-03-18T20:24:45.442127+0000","last_fullsized":"2024-03-18T20:24:45.442127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:25:15.699466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962679+0000","last_change":"2024-03-18T20:24:45.445996+0000","last_active":"2024-03-18T20:24:45.962679+0000","last_peered":"2024-03-18T20:24:45.962679+0000","last_clean":"2024-03-18T20:24:45.962679+0000","last_became_active":"2024-03-18T20:24:45.445552+0000","last_became_peered":"2024-03-18T20:24:45.445552+0000","last_unstale":"2024-03-18T20:24:45.962679+0000","last_undegraded":"2024-03-18T20:24:45.962679+0000","last_fullsized":"2024-03-18T20:24:45.962679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:26.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476209+0000","last_change":"2024-03-18T20:24:40.951499+0000","last_active":"2024-03-18T20:24:59.476209+0000","last_peered":"2024-03-18T20:24:59.476209+0000","last_clean":"2024-03-18T20:24:59.476209+0000","last_became_active":"2024-03-18T20:24:40.951183+0000","last_became_peered":"2024-03-18T20:24:40.951183+0000","last_unstale":"2024-03-18T20:24:59.476209+0000","last_undegraded":"2024-03-18T20:24:59.476209+0000","last_fullsized":"2024-03-18T20:24:59.476209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:25.936370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432254+0000","last_change":"2024-03-18T20:24:41.354835+0000","last_active":"2024-03-18T20:24:45.432254+0000","last_peered":"2024-03-18T20:24:45.432254+0000","last_clean":"2024-03-18T20:24:45.432254+0000","last_became_active":"2024-03-18T20:24:41.354486+0000","last_became_peered":"2024-03-18T20:24:41.354486+0000","last_unstale":"2024-03-18T20:24:45.432254+0000","last_undegraded":"2024-03-18T20:24:45.432254+0000","last_fullsized":"2024-03-18T20:24:45.432254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961610+0000","last_change":"2024-03-18T20:24:45.446772+0000","last_active":"2024-03-18T20:24:45.961610+0000","last_peered":"2024-03-18T20:24:45.961610+0000","last_clean":"2024-03-18T20:24:45.961610+0000","last_became_active":"2024-03-18T20:24:45.446296+0000","last_became_peered":"2024-03-18T20:24:45.446296+0000","last_unstale":"2024-03-18T20:24:45.961610+0000","last_undegraded":"2024-03-18T20:24:45.961610+0000","last_fullsized":"2024-03-18T20:24:45.961610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:41:20.784772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431809+0000","last_change":"2024-03-18T20:24:40.948077+0000","last_active":"2024-03-18T20:24:45.431809+0000","last_peered":"2024-03-18T20:24:45.431809+0000","last_clean":"2024-03-18T20:24:45.431809+0000","last_became_active":"2024-03-18T20:24:40.946970+0000","last_became_peered":"2024-03-18T20:24:40.946970+0000","last_unstale":"2024-03-18T20:24:45.431809+0000","last_undegraded":"2024-03-18T20:24:45.431809+0000","last_fullsized":"2024-03-18T20:24:45.431809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:34.558466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437521+0000","last_change":"2024-03-18T20:24:41.371040+0000","last_active":"2024-03-18T20:24:45.437521+0000","last_peered":"2024-03-18T20:24:45.437521+0000","last_clean":"2024-03-18T20:24:45.437521+0000","last_became_active":"2024-03-18T20:24:41.370654+0000","last_became_peered":"2024-03-18T20:24:41.370654+0000","last_unstale":"2024-03-18T20:24:45.437521+0000","last_undegraded":"2024-03-18T20:24:45.437521+0000","last_fullsized":"2024-03-18T20:24:45.437521+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854817+0000","last_change":"2024-03-18T20:24:45.438825+0000","last_active":"2024-03-18T20:24:45.854817+0000","last_peered":"2024-03-18T20:24:45.854817+0000","last_clean":"2024-03-18T20:24:45.854817+0000","last_became_active":"2024-03-18T20:24:45.438418+0000","last_became_peered":"2024-03-18T20:24:45.438418+0000","last_unstale":"2024-03-18T20:24:45.854817+0000","last_undegraded":"2024-03-18T20:24:45.854817+0000","last_fullsized":"2024-03-18T20:24:45.854817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:52.562628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472147+0000","last_change":"2024-03-18T20:24:40.941503+0000","last_active":"2024-03-18T20:24:58.472147+0000","last_peered":"2024-03-18T20:24:58.472147+0000","last_clean":"2024-03-18T20:24:58.472147+0000","last_became_active":"2024-03-18T20:24:40.941150+0000","last_became_peered":"2024-03-18T20:24:40.941150+0000","last_unstale":"2024-03-18T20:24:58.472147+0000","last_undegraded":"2024-03-18T20:24:58.472147+0000","last_fullsized":"2024-03-18T20:24:58.472147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:02.509071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430556+0000","last_change":"2024-03-18T20:24:41.369688+0000","last_active":"2024-03-18T20:24:45.430556+0000","last_peered":"2024-03-18T20:24:45.430556+0000","last_clean":"2024-03-18T20:24:45.430556+0000","last_became_active":"2024-03-18T20:24:41.369361+0000","last_became_peered":"2024-03-18T20:24:41.369361+0000","last_unstale":"2024-03-18T20:24:45.430556+0000","last_undegraded":"2024-03-18T20:24:45.430556+0000","last_fullsized":"2024-03-18T20:24:45.430556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:23.128929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854843+0000","last_change":"2024-03-18T20:24:45.440139+0000","last_active":"2024-03-18T20:24:45.854843+0000","last_peered":"2024-03-18T20:24:45.854843+0000","last_clean":"2024-03-18T20:24:45.854843+0000","last_became_active":"2024-03-18T20:24:45.439758+0000","last_became_peered":"2024-03-18T20:24:45.439758+0000","last_unstale":"2024-03-18T20:24:45.854843+0000","last_undegraded":"2024-03-18T20:24:45.854843+0000","last_fullsized":"2024-03-18T20:24:45.854843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:57.064268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476530+0000","last_change":"2024-03-18T20:24:40.944915+0000","last_active":"2024-03-18T20:24:59.476530+0000","last_peered":"2024-03-18T20:24:59.476530+0000","last_clean":"2024-03-18T20:24:59.476530+0000","last_became_active":"2024-03-18T20:24:40.944547+0000","last_became_peered":"2024-03-18T20:24:40.944547+0000","last_unstale":"2024-03-18T20:24:59.476530+0000","last_undegraded":"2024-03-18T20:24:59.476530+0000","last_fullsized":"2024-03-18T20:24:59.476530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:36.732318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437751+0000","last_change":"2024-03-18T20:24:41.371113+0000","last_active":"2024-03-18T20:24:45.437751+0000","last_peered":"2024-03-18T20:24:45.437751+0000","last_clean":"2024-03-18T20:24:45.437751+0000","last_became_active":"2024-03-18T20:24:41.370764+0000","last_became_peered":"2024-03-18T20:24:41.370764+0000","last_unstale":"2024-03-18T20:24:45.437751+0000","last_undegraded":"2024-03-18T20:24:45.437751+0000","last_fullsized":"2024-03-18T20:24:45.437751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:51.181585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449570+0000","last_change":"2024-03-18T20:24:45.444798+0000","last_active":"2024-03-18T20:24:45.449570+0000","last_peered":"2024-03-18T20:24:45.449570+0000","last_clean":"2024-03-18T20:24:45.449570+0000","last_became_active":"2024-03-18T20:24:45.444374+0000","last_became_peered":"2024-03-18T20:24:45.444374+0000","last_unstale":"2024-03-18T20:24:45.449570+0000","last_undegraded":"2024-03-18T20:24:45.449570+0000","last_fullsized":"2024-03-18T20:24:45.449570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:54.055384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429724+0000","last_change":"2024-03-18T20:24:40.936710+0000","last_active":"2024-03-18T20:24:45.429724+0000","last_peered":"2024-03-18T20:24:45.429724+0000","last_clean":"2024-03-18T20:24:45.429724+0000","last_became_active":"2024-03-18T20:24:40.936360+0000","last_became_peered":"2024-03-18T20:24:40.936360+0000","last_unstale":"2024-03-18T20:24:45.429724+0000","last_undegraded":"2024-03-18T20:24:45.429724+0000","last_fullsized":"2024-03-18T20:24:45.429724+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:08.794520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425954+0000","last_change":"2024-03-18T20:24:41.355109+0000","last_active":"2024-03-18T20:24:45.425954+0000","last_peered":"2024-03-18T20:24:45.425954+0000","last_clean":"2024-03-18T20:24:45.425954+0000","last_became_active":"2024-03-18T20:24:41.354718+0000","last_became_peered":"2024-03-18T20:24:41.354718+0000","last_unstale":"2024-03-18T20:24:45.425954+0000","last_undegraded":"2024-03-18T20:24:45.425954+0000","last_fullsized":"2024-03-18T20:24:45.425954+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:39.147267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963393+0000","last_change":"2024-03-18T20:24:45.451275+0000","last_active":"2024-03-18T20:24:45.963393+0000","last_peered":"2024-03-18T20:24:45.963393+0000","last_clean":"2024-03-18T20:24:45.963393+0000","last_became_active":"2024-03-18T20:24:45.450883+0000","last_became_peered":"2024-03-18T20:24:45.450883+0000","last_unstale":"2024-03-18T20:24:45.963393+0000","last_undegraded":"2024-03-18T20:24:45.963393+0000","last_fullsized":"2024-03-18T20:24:45.963393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:10:41.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961539+0000","last_change":"2024-03-18T20:24:40.939692+0000","last_active":"2024-03-18T20:24:45.961539+0000","last_peered":"2024-03-18T20:24:45.961539+0000","last_clean":"2024-03-18T20:24:45.961539+0000","last_became_active":"2024-03-18T20:24:40.939363+0000","last_became_peered":"2024-03-18T20:24:40.939363+0000","last_unstale":"2024-03-18T20:24:45.961539+0000","last_undegraded":"2024-03-18T20:24:45.961539+0000","last_fullsized":"2024-03-18T20:24:45.961539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:41.445337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430758+0000","last_change":"2024-03-18T20:24:41.357835+0000","last_active":"2024-03-18T20:24:45.430758+0000","last_peered":"2024-03-18T20:24:45.430758+0000","last_clean":"2024-03-18T20:24:45.430758+0000","last_became_active":"2024-03-18T20:24:41.357534+0000","last_became_peered":"2024-03-18T20:24:41.357534+0000","last_unstale":"2024-03-18T20:24:45.430758+0000","last_undegraded":"2024-03-18T20:24:45.430758+0000","last_fullsized":"2024-03-18T20:24:45.430758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:16.662257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440398+0000","last_change":"2024-03-18T20:24:45.438295+0000","last_active":"2024-03-18T20:24:45.440398+0000","last_peered":"2024-03-18T20:24:45.440398+0000","last_clean":"2024-03-18T20:24:45.440398+0000","last_became_active":"2024-03-18T20:24:45.437955+0000","last_became_peered":"2024-03-18T20:24:45.437955+0000","last_unstale":"2024-03-18T20:24:45.440398+0000","last_undegraded":"2024-03-18T20:24:45.440398+0000","last_fullsized":"2024-03-18T20:24:45.440398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:26.435144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472214+0000","last_change":"2024-03-18T20:24:40.945507+0000","last_active":"2024-03-18T20:24:58.472214+0000","last_peered":"2024-03-18T20:24:58.472214+0000","last_clean":"2024-03-18T20:24:58.472214+0000","last_became_active":"2024-03-18T20:24:40.945195+0000","last_became_peered":"2024-03-18T20:24:40.945195+0000","last_unstale":"2024-03-18T20:24:58.472214+0000","last_undegraded":"2024-03-18T20:24:58.472214+0000","last_fullsized":"2024-03-18T20:24:58.472214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:29.173130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431923+0000","last_change":"2024-03-18T20:24:41.356795+0000","last_active":"2024-03-18T20:24:45.431923+0000","last_peered":"2024-03-18T20:24:45.431923+0000","last_clean":"2024-03-18T20:24:45.431923+0000","last_became_active":"2024-03-18T20:24:41.356461+0000","last_became_peered":"2024-03-18T20:24:41.356461+0000","last_unstale":"2024-03-18T20:24:45.431923+0000","last_undegraded":"2024-03-18T20:24:45.431923+0000","last_fullsized":"2024-03-18T20:24:45.431923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:04.230985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444215+0000","last_change":"2024-03-18T20:24:45.439984+0000","last_active":"2024-03-18T20:24:45.444215+0000","last_peered":"2024-03-18T20:24:45.444215+0000","last_clean":"2024-03-18T20:24:45.444215+0000","last_became_active":"2024-03-18T20:24:45.439598+0000","last_became_peered":"2024-03-18T20:24:45.439598+0000","last_unstale":"2024-03-18T20:24:45.444215+0000","last_undegraded":"2024-03-18T20:24:45.444215+0000","last_fullsized":"2024-03-18T20:24:45.444215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:49.481812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469645+0000","last_change":"2024-03-18T20:24:40.943978+0000","last_active":"2024-03-18T20:24:58.469645+0000","last_peered":"2024-03-18T20:24:58.469645+0000","last_clean":"2024-03-18T20:24:58.469645+0000","last_became_active":"2024-03-18T20:24:40.943623+0000","last_became_peered":"2024-03-18T20:24:40.943623+0000","last_unstale":"2024-03-18T20:24:58.469645+0000","last_undegraded":"2024-03-18T20:24:58.469645+0000","last_fullsized":"2024-03-18T20:24:58.469645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:58.995256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442197+0000","last_change":"2024-03-18T20:24:41.365465+0000","last_active":"2024-03-18T20:24:45.442197+0000","last_peered":"2024-03-18T20:24:45.442197+0000","last_clean":"2024-03-18T20:24:45.442197+0000","last_became_active":"2024-03-18T20:24:41.365146+0000","last_became_peered":"2024-03-18T20:24:41.365146+0000","last_unstale":"2024-03-18T20:24:45.442197+0000","last_undegraded":"2024-03-18T20:24:45.442197+0000","last_fullsized":"2024-03-18T20:24:45.442197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:47.903106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962967+0000","last_change":"2024-03-18T20:24:45.440065+0000","last_active":"2024-03-18T20:24:45.962967+0000","last_peered":"2024-03-18T20:24:45.962967+0000","last_clean":"2024-03-18T20:24:45.962967+0000","last_became_active":"2024-03-18T20:24:45.439424+0000","last_became_peered":"2024-03-18T20:24:45.439424+0000","last_unstale":"2024-03-18T20:24:45.962967+0000","last_undegraded":"2024-03-18T20:24:45.962967+0000","last_fullsized":"2024-03-18T20:24:45.962967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:28.350851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961988+0000","last_change":"2024-03-18T20:24:40.942424+0000","last_active":"2024-03-18T20:24:45.961988+0000","last_peered":"2024-03-18T20:24:45.961988+0000","last_clean":"2024-03-18T20:24:45.961988+0000","last_became_active":"2024-03-18T20:24:40.941878+0000","last_became_peered":"2024-03-18T20:24:40.941878+0000","last_unstale":"2024-03-18T20:24:45.961988+0000","last_undegraded":"2024-03-18T20:24:45.961988+0000","last_fullsized":"2024-03-18T20:24:45.961988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:40.746350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432427+0000","last_change":"2024-03-18T20:24:41.372684+0000","last_active":"2024-03-18T20:24:45.432427+0000","last_peered":"2024-03-18T20:24:45.432427+0000","last_clean":"2024-03-18T20:24:45.432427+0000","last_became_active":"2024-03-18T20:24:41.372171+0000","last_became_peered":"2024-03-18T20:24:41.372171+0000","last_unstale":"2024-03-18T20:24:45.432427+0000","last_undegraded":"2024-03-18T20:24:45.432427+0000","last_fullsized":"2024-03-18T20:24:45.432427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:50.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444306+0000","last_change":"2024-03-18T20:24:45.440289+0000","last_active":"2024-03-18T20:24:45.444306+0000","last_peered":"2024-03-18T20:24:45.444306+0000","last_clean":"2024-03-18T20:24:45.444306+0000","last_became_active":"2024-03-18T20:24:45.439910+0000","last_became_peered":"2024-03-18T20:24:45.439910+0000","last_unstale":"2024-03-18T20:24:45.444306+0000","last_undegraded":"2024-03-18T20:24:45.444306+0000","last_fullsized":"2024-03-18T20:24:45.444306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:19.420493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.487757+0000","last_change":"2024-03-18T20:24:40.943843+0000","last_active":"2024-03-18T20:25:00.487757+0000","last_peered":"2024-03-18T20:25:00.487757+0000","last_clean":"2024-03-18T20:25:00.487757+0000","last_became_active":"2024-03-18T20:24:40.943521+0000","last_became_peered":"2024-03-18T20:24:40.943521+0000","last_unstale":"2024-03-18T20:25:00.487757+0000","last_undegraded":"2024-03-18T20:25:00.487757+0000","last_fullsized":"2024-03-18T20:25:00.487757+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:44.872498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433854+0000","last_change":"2024-03-18T20:24:41.358139+0000","last_active":"2024-03-18T20:24:45.433854+0000","last_peered":"2024-03-18T20:24:45.433854+0000","last_clean":"2024-03-18T20:24:45.433854+0000","last_became_active":"2024-03-18T20:24:41.357789+0000","last_became_peered":"2024-03-18T20:24:41.357789+0000","last_unstale":"2024-03-18T20:24:45.433854+0000","last_undegraded":"2024-03-18T20:24:45.433854+0000","last_fullsized":"2024-03-18T20:24:45.433854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:49.643210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449427+0000","last_change":"2024-03-18T20:24:45.448899+0000","last_active":"2024-03-18T20:24:45.449427+0000","last_peered":"2024-03-18T20:24:45.449427+0000","last_clean":"2024-03-18T20:24:45.449427+0000","last_became_active":"2024-03-18T20:24:45.448519+0000","last_became_peered":"2024-03-18T20:24:45.448519+0000","last_unstale":"2024-03-18T20:24:45.449427+0000","last_undegraded":"2024-03-18T20:24:45.449427+0000","last_fullsized":"2024-03-18T20:24:45.449427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:54.071495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469997+0000","last_change":"2024-03-18T20:24:40.944455+0000","last_active":"2024-03-18T20:24:58.469997+0000","last_peered":"2024-03-18T20:24:58.469997+0000","last_clean":"2024-03-18T20:24:58.469997+0000","last_became_active":"2024-03-18T20:24:40.944115+0000","last_became_peered":"2024-03-18T20:24:40.944115+0000","last_unstale":"2024-03-18T20:24:58.469997+0000","last_undegraded":"2024-03-18T20:24:58.469997+0000","last_fullsized":"2024-03-18T20:24:58.469997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:06.160375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430066+0000","last_change":"2024-03-18T20:24:41.356074+0000","last_active":"2024-03-18T20:24:45.430066+0000","last_peered":"2024-03-18T20:24:45.430066+0000","last_clean":"2024-03-18T20:24:45.430066+0000","last_became_active":"2024-03-18T20:24:41.355755+0000","last_became_peered":"2024-03-18T20:24:41.355755+0000","last_unstale":"2024-03-18T20:24:45.430066+0000","last_undegraded":"2024-03-18T20:24:45.430066+0000","last_fullsized":"2024-03-18T20:24:45.430066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:54.007791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.443111+0000","last_change":"2024-03-18T20:24:45.440797+0000","last_active":"2024-03-18T20:24:45.443111+0000","last_peered":"2024-03-18T20:24:45.443111+0000","last_clean":"2024-03-18T20:24:45.443111+0000","last_became_active":"2024-03-18T20:24:45.440410+0000","last_became_peered":"2024-03-18T20:24:45.440410+0000","last_unstale":"2024-03-18T20:24:45.443111+0000","last_undegraded":"2024-03-18T20:24:45.443111+0000","last_fullsized":"2024-03-18T20:24:45.443111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:02.586794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471938+0000","last_change":"2024-03-18T20:24:40.944239+0000","last_active":"2024-03-18T20:24:58.471938+0000","last_peered":"2024-03-18T20:24:58.471938+0000","last_clean":"2024-03-18T20:24:58.471938+0000","last_became_active":"2024-03-18T20:24:40.943845+0000","last_became_peered":"2024-03-18T20:24:40.943845+0000","last_unstale":"2024-03-18T20:24:58.471938+0000","last_undegraded":"2024-03-18T20:24:58.471938+0000","last_fullsized":"2024-03-18T20:24:58.471938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:30.966789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436700+0000","last_change":"2024-03-18T20:24:41.373565+0000","last_active":"2024-03-18T20:24:45.436700+0000","last_peered":"2024-03-18T20:24:45.436700+0000","last_clean":"2024-03-18T20:24:45.436700+0000","last_became_active":"2024-03-18T20:24:41.373201+0000","last_became_peered":"2024-03-18T20:24:41.373201+0000","last_unstale":"2024-03-18T20:24:45.436700+0000","last_undegraded":"2024-03-18T20:24:45.436700+0000","last_fullsized":"2024-03-18T20:24:45.436700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:13.118518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.447664+0000","last_change":"2024-03-18T20:24:45.447115+0000","last_active":"2024-03-18T20:24:45.447664+0000","last_peered":"2024-03-18T20:24:45.447664+0000","last_clean":"2024-03-18T20:24:45.447664+0000","last_became_active":"2024-03-18T20:24:45.446316+0000","last_became_peered":"2024-03-18T20:24:45.446316+0000","last_unstale":"2024-03-18T20:24:45.447664+0000","last_undegraded":"2024-03-18T20:24:45.447664+0000","last_fullsized":"2024-03-18T20:24:45.447664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:17.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433430+0000","last_change":"2024-03-18T20:24:40.938678+0000","last_active":"2024-03-18T20:24:45.433430+0000","last_peered":"2024-03-18T20:24:45.433430+0000","last_clean":"2024-03-18T20:24:45.433430+0000","last_became_active":"2024-03-18T20:24:40.938300+0000","last_became_peered":"2024-03-18T20:24:40.938300+0000","last_unstale":"2024-03-18T20:24:45.433430+0000","last_undegraded":"2024-03-18T20:24:45.433430+0000","last_fullsized":"2024-03-18T20:24:45.433430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:21.806721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434871+0000","last_change":"2024-03-18T20:24:41.368977+0000","last_active":"2024-03-18T20:24:45.434871+0000","last_peered":"2024-03-18T20:24:45.434871+0000","last_clean":"2024-03-18T20:24:45.434871+0000","last_became_active":"2024-03-18T20:24:41.368545+0000","last_became_peered":"2024-03-18T20:24:41.368545+0000","last_unstale":"2024-03-18T20:24:45.434871+0000","last_undegraded":"2024-03-18T20:24:45.434871+0000","last_fullsized":"2024-03-18T20:24:45.434871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:23.827215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962771+0000","last_change":"2024-03-18T20:24:45.445759+0000","last_active":"2024-03-18T20:24:45.962771+0000","last_peered":"2024-03-18T20:24:45.962771+0000","last_clean":"2024-03-18T20:24:45.962771+0000","last_became_active":"2024-03-18T20:24:45.445366+0000","last_became_peered":"2024-03-18T20:24:45.445366+0000","last_unstale":"2024-03-18T20:24:45.962771+0000","last_undegraded":"2024-03-18T20:24:45.962771+0000","last_fullsized":"2024-03-18T20:24:45.962771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:59.103814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479110+0000","last_change":"2024-03-18T20:24:40.950980+0000","last_active":"2024-03-18T20:24:59.479110+0000","last_peered":"2024-03-18T20:24:59.479110+0000","last_clean":"2024-03-18T20:24:59.479110+0000","last_became_active":"2024-03-18T20:24:40.950280+0000","last_became_peered":"2024-03-18T20:24:40.950280+0000","last_unstale":"2024-03-18T20:24:59.479110+0000","last_undegraded":"2024-03-18T20:24:59.479110+0000","last_fullsized":"2024-03-18T20:24:59.479110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:16:08.679066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437457+0000","last_change":"2024-03-18T20:24:41.374302+0000","last_active":"2024-03-18T20:24:45.437457+0000","last_peered":"2024-03-18T20:24:45.437457+0000","last_clean":"2024-03-18T20:24:45.437457+0000","last_became_active":"2024-03-18T20:24:41.373974+0000","last_became_peered":"2024-03-18T20:24:41.373974+0000","last_unstale":"2024-03-18T20:24:45.437457+0000","last_undegraded":"2024-03-18T20:24:45.437457+0000","last_fullsized":"2024-03-18T20:24:45.437457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445064+0000","last_change":"2024-03-18T20:24:45.443527+0000","last_active":"2024-03-18T20:24:45.445064+0000","last_peered":"2024-03-18T20:24:45.445064+0000","last_clean":"2024-03-18T20:24:45.445064+0000","last_became_active":"2024-03-18T20:24:45.443105+0000","last_became_peered":"2024-03-18T20:24:45.443105+0000","last_unstale":"2024-03-18T20:24:45.445064+0000","last_undegraded":"2024-03-18T20:24:45.445064+0000","last_fullsized":"2024-03-18T20:24:45.445064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:04.568061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962307+0000","last_change":"2024-03-18T20:24:40.948223+0000","last_active":"2024-03-18T20:24:45.962307+0000","last_peered":"2024-03-18T20:24:45.962307+0000","last_clean":"2024-03-18T20:24:45.962307+0000","last_became_active":"2024-03-18T20:24:40.947889+0000","last_became_peered":"2024-03-18T20:24:40.947889+0000","last_unstale":"2024-03-18T20:24:45.962307+0000","last_undegraded":"2024-03-18T20:24:45.962307+0000","last_fullsized":"2024-03-18T20:24:45.962307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:33.804508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430232+0000","last_change":"2024-03-18T20:24:41.369441+0000","last_active":"2024-03-18T20:24:45.430232+0000","last_peered":"2024-03-18T20:24:45.430232+0000","last_clean":"2024-03-18T20:24:45.430232+0000","last_became_active":"2024-03-18T20:24:41.368271+0000","last_became_peered":"2024-03-18T20:24:41.368271+0000","last_unstale":"2024-03-18T20:24:45.430232+0000","last_undegraded":"2024-03-18T20:24:45.430232+0000","last_fullsized":"2024-03-18T20:24:45.430232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:20.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962427+0000","last_change":"2024-03-18T20:24:45.441117+0000","last_active":"2024-03-18T20:24:45.962427+0000","last_peered":"2024-03-18T20:24:45.962427+0000","last_clean":"2024-03-18T20:24:45.962427+0000","last_became_active":"2024-03-18T20:24:45.440774+0000","last_became_peered":"2024-03-18T20:24:45.440774+0000","last_unstale":"2024-03-18T20:24:45.962427+0000","last_undegraded":"2024-03-18T20:24:45.962427+0000","last_fullsized":"2024-03-18T20:24:45.962427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:27.034432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431547+0000","last_change":"2024-03-18T20:24:40.940987+0000","last_active":"2024-03-18T20:24:45.431547+0000","last_peered":"2024-03-18T20:24:45.431547+0000","last_clean":"2024-03-18T20:24:45.431547+0000","last_became_active":"2024-03-18T20:24:40.940652+0000","last_became_peered":"2024-03-18T20:24:40.940652+0000","last_unstale":"2024-03-18T20:24:45.431547+0000","last_undegraded":"2024-03-18T20:24:45.431547+0000","last_fullsized":"2024-03-18T20:24:45.431547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:57.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852466+0000","last_change":"2024-03-18T20:24:41.356381+0000","last_active":"2024-03-18T20:24:45.852466+0000","last_peered":"2024-03-18T20:24:45.852466+0000","last_clean":"2024-03-18T20:24:45.852466+0000","last_became_active":"2024-03-18T20:24:41.355078+0000","last_became_peered":"2024-03-18T20:24:41.355078+0000","last_unstale":"2024-03-18T20:24:45.852466+0000","last_undegraded":"2024-03-18T20:24:45.852466+0000","last_fullsized":"2024-03-18T20:24:45.852466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:03.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961701+0000","last_change":"2024-03-18T20:24:45.448305+0000","last_active":"2024-03-18T20:24:45.961701+0000","last_peered":"2024-03-18T20:24:45.961701+0000","last_clean":"2024-03-18T20:24:45.961701+0000","last_became_active":"2024-03-18T20:24:45.447706+0000","last_became_peered":"2024-03-18T20:24:45.447706+0000","last_unstale":"2024-03-18T20:24:45.961701+0000","last_undegraded":"2024-03-18T20:24:45.961701+0000","last_fullsized":"2024-03-18T20:24:45.961701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:22.305139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963464+0000","last_change":"2024-03-18T20:24:40.939482+0000","last_active":"2024-03-18T20:24:45.963464+0000","last_peered":"2024-03-18T20:24:45.963464+0000","last_clean":"2024-03-18T20:24:45.963464+0000","last_became_active":"2024-03-18T20:24:40.938983+0000","last_became_peered":"2024-03-18T20:24:40.938983+0000","last_unstale":"2024-03-18T20:24:45.963464+0000","last_undegraded":"2024-03-18T20:24:45.963464+0000","last_fullsized":"2024-03-18T20:24:45.963464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:00.111633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852446+0000","last_change":"2024-03-18T20:24:41.366749+0000","last_active":"2024-03-18T20:24:45.852446+0000","last_peered":"2024-03-18T20:24:45.852446+0000","last_clean":"2024-03-18T20:24:45.852446+0000","last_became_active":"2024-03-18T20:24:41.366335+0000","last_became_peered":"2024-03-18T20:24:41.366335+0000","last_unstale":"2024-03-18T20:24:45.852446+0000","last_undegraded":"2024-03-18T20:24:45.852446+0000","last_fullsized":"2024-03-18T20:24:45.852446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:36.927348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962802+0000","last_change":"2024-03-18T20:24:45.447585+0000","last_active":"2024-03-18T20:24:45.962802+0000","last_peered":"2024-03-18T20:24:45.962802+0000","last_clean":"2024-03-18T20:24:45.962802+0000","last_became_active":"2024-03-18T20:24:45.447165+0000","last_became_peered":"2024-03-18T20:24:45.447165+0000","last_unstale":"2024-03-18T20:24:45.962802+0000","last_undegraded":"2024-03-18T20:24:45.962802+0000","last_fullsized":"2024-03-18T20:24:45.962802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:14:34.960675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438282+0000","last_change":"2024-03-18T20:24:41.371969+0000","last_active":"2024-03-18T20:24:45.438282+0000","last_peered":"2024-03-18T20:24:45.438282+0000","last_clean":"2024-03-18T20:24:45.438282+0000","last_became_active":"2024-03-18T20:24:41.371516+0000","last_became_peered":"2024-03-18T20:24:41.371516+0000","last_unstale":"2024-03-18T20:24:45.438282+0000","last_undegraded":"2024-03-18T20:24:45.438282+0000","last_fullsized":"2024-03-18T20:24:45.438282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:46.761356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429629+0000","last_change":"2024-03-18T20:24:40.940375+0000","last_active":"2024-03-18T20:24:45.429629+0000","last_peered":"2024-03-18T20:24:45.429629+0000","last_clean":"2024-03-18T20:24:45.429629+0000","last_became_active":"2024-03-18T20:24:40.940053+0000","last_became_peered":"2024-03-18T20:24:40.940053+0000","last_unstale":"2024-03-18T20:24:45.429629+0000","last_undegraded":"2024-03-18T20:24:45.429629+0000","last_fullsized":"2024-03-18T20:24:45.429629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:20:17.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444410+0000","last_change":"2024-03-18T20:24:45.443315+0000","last_active":"2024-03-18T20:24:45.444410+0000","last_peered":"2024-03-18T20:24:45.444410+0000","last_clean":"2024-03-18T20:24:45.444410+0000","last_became_active":"2024-03-18T20:24:45.442889+0000","last_became_peered":"2024-03-18T20:24:45.442889+0000","last_unstale":"2024-03-18T20:24:45.444410+0000","last_undegraded":"2024-03-18T20:24:45.444410+0000","last_fullsized":"2024-03-18T20:24:45.444410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:40.097397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425997+0000","last_change":"2024-03-18T20:24:41.368952+0000","last_active":"2024-03-18T20:24:45.425997+0000","last_peered":"2024-03-18T20:24:45.425997+0000","last_clean":"2024-03-18T20:24:45.425997+0000","last_became_active":"2024-03-18T20:24:41.368629+0000","last_became_peered":"2024-03-18T20:24:41.368629+0000","last_unstale":"2024-03-18T20:24:45.425997+0000","last_undegraded":"2024-03-18T20:24:45.425997+0000","last_fullsized":"2024-03-18T20:24:45.425997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:28:58.521712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432237+0000","last_change":"2024-03-18T20:24:40.938841+0000","last_active":"2024-03-18T20:24:45.432237+0000","last_peered":"2024-03-18T20:24:45.432237+0000","last_clean":"2024-03-18T20:24:45.432237+0000","last_became_active":"2024-03-18T20:24:40.938464+0000","last_became_peered":"2024-03-18T20:24:40.938464+0000","last_unstale":"2024-03-18T20:24:45.432237+0000","last_undegraded":"2024-03-18T20:24:45.432237+0000","last_fullsized":"2024-03-18T20:24:45.432237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:04.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":325,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434656+0000","last_change":"2024-03-18T20:23:54.597665+0000","last_active":"2024-03-18T20:24:45.434656+0000","last_peered":"2024-03-18T20:24:45.434656+0000","last_clean":"2024-03-18T20:24:45.434656+0000","last_became_active":"2024-03-18T20:23:54.597135+0000","last_became_peered":"2024-03-18T20:23:54.597135+0000","last_unstale":"2024-03-18T20:24:45.434656+0000","last_undegraded":"2024-03-18T20:24:45.434656+0000","last_fullsized":"2024-03-18T20:24:45.434656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_clean_scrub_stamp":"2024-03-18T20:21:34.333460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:59.546138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449625+0000","last_change":"2024-03-18T20:24:45.448945+0000","last_active":"2024-03-18T20:24:45.449625+0000","last_peered":"2024-03-18T20:24:45.449625+0000","last_clean":"2024-03-18T20:24:45.449625+0000","last_became_active":"2024-03-18T20:24:45.448585+0000","last_became_peered":"2024-03-18T20:24:45.448585+0000","last_unstale":"2024-03-18T20:24:45.449625+0000","last_undegraded":"2024-03-18T20:24:45.449625+0000","last_fullsized":"2024-03-18T20:24:45.449625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:49.668118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429530+0000","last_change":"2024-03-18T20:24:41.364453+0000","last_active":"2024-03-18T20:24:45.429530+0000","last_peered":"2024-03-18T20:24:45.429530+0000","last_clean":"2024-03-18T20:24:45.429530+0000","last_became_active":"2024-03-18T20:24:41.364047+0000","last_became_peered":"2024-03-18T20:24:41.364047+0000","last_unstale":"2024-03-18T20:24:45.429530+0000","last_undegraded":"2024-03-18T20:24:45.429530+0000","last_fullsized":"2024-03-18T20:24:45.429530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:32.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476629+0000","last_change":"2024-03-18T20:24:40.939926+0000","last_active":"2024-03-18T20:24:59.476629+0000","last_peered":"2024-03-18T20:24:59.476629+0000","last_clean":"2024-03-18T20:24:59.476629+0000","last_became_active":"2024-03-18T20:24:40.939514+0000","last_became_peered":"2024-03-18T20:24:40.939514+0000","last_unstale":"2024-03-18T20:24:59.476629+0000","last_undegraded":"2024-03-18T20:24:59.476629+0000","last_fullsized":"2024-03-18T20:24:59.476629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:41.852567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442992+0000","last_change":"2024-03-18T20:24:45.439880+0000","last_active":"2024-03-18T20:24:45.442992+0000","last_peered":"2024-03-18T20:24:45.442992+0000","last_clean":"2024-03-18T20:24:45.442992+0000","last_became_active":"2024-03-18T20:24:45.439455+0000","last_became_peered":"2024-03-18T20:24:45.439455+0000","last_unstale":"2024-03-18T20:24:45.442992+0000","last_undegraded":"2024-03-18T20:24:45.442992+0000","last_fullsized":"2024-03-18T20:24:45.442992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:26.050401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470311+0000","last_change":"2024-03-18T20:24:40.947764+0000","last_active":"2024-03-18T20:24:58.470311+0000","last_peered":"2024-03-18T20:24:58.470311+0000","last_clean":"2024-03-18T20:24:58.470311+0000","last_became_active":"2024-03-18T20:24:40.946947+0000","last_became_peered":"2024-03-18T20:24:40.946947+0000","last_unstale":"2024-03-18T20:24:58.470311+0000","last_undegraded":"2024-03-18T20:24:58.470311+0000","last_fullsized":"2024-03-18T20:24:58.470311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:38.279134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431036+0000","last_change":"2024-03-18T20:24:41.352213+0000","last_active":"2024-03-18T20:24:45.431036+0000","last_peered":"2024-03-18T20:24:45.431036+0000","last_clean":"2024-03-18T20:24:45.431036+0000","last_became_active":"2024-03-18T20:24:41.351895+0000","last_became_peered":"2024-03-18T20:24:41.351895+0000","last_unstale":"2024-03-18T20:24:45.431036+0000","last_undegraded":"2024-03-18T20:24:45.431036+0000","last_fullsized":"2024-03-18T20:24:45.431036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:55.756782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440641+0000","last_change":"2024-03-18T20:24:45.439483+0000","last_active":"2024-03-18T20:24:45.440641+0000","last_peered":"2024-03-18T20:24:45.440641+0000","last_clean":"2024-03-18T20:24:45.440641+0000","last_became_active":"2024-03-18T20:24:45.436952+0000","last_became_peered":"2024-03-18T20:24:45.436952+0000","last_unstale":"2024-03-18T20:24:45.440641+0000","last_undegraded":"2024-03-18T20:24:45.440641+0000","last_fullsized":"2024-03-18T20:24:45.440641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:33.839948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.478344+0000","last_change":"2024-03-18T20:24:40.948184+0000","last_active":"2024-03-18T20:24:59.478344+0000","last_peered":"2024-03-18T20:24:59.478344+0000","last_clean":"2024-03-18T20:24:59.478344+0000","last_became_active":"2024-03-18T20:24:40.947953+0000","last_became_peered":"2024-03-18T20:24:40.947953+0000","last_unstale":"2024-03-18T20:24:59.478344+0000","last_undegraded":"2024-03-18T20:24:59.478344+0000","last_fullsized":"2024-03-18T20:24:59.478344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:51.701653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426390+0000","last_change":"2024-03-18T20:24:41.356563+0000","last_active":"2024-03-18T20:24:45.426390+0000","last_peered":"2024-03-18T20:24:45.426390+0000","last_clean":"2024-03-18T20:24:45.426390+0000","last_became_active":"2024-03-18T20:24:41.356229+0000","last_became_peered":"2024-03-18T20:24:41.356229+0000","last_unstale":"2024-03-18T20:24:45.426390+0000","last_undegraded":"2024-03-18T20:24:45.426390+0000","last_fullsized":"2024-03-18T20:24:45.426390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:20.296131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854824+0000","last_change":"2024-03-18T20:24:45.440937+0000","last_active":"2024-03-18T20:24:45.854824+0000","last_peered":"2024-03-18T20:24:45.854824+0000","last_clean":"2024-03-18T20:24:45.854824+0000","last_became_active":"2024-03-18T20:24:45.440624+0000","last_became_peered":"2024-03-18T20:24:45.440624+0000","last_unstale":"2024-03-18T20:24:45.854824+0000","last_undegraded":"2024-03-18T20:24:45.854824+0000","last_fullsized":"2024-03-18T20:24:45.854824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:41.588153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"80'2","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430298+0000","last_change":"2024-03-18T20:24:40.940622+0000","last_active":"2024-03-18T20:24:45.430298+0000","last_peered":"2024-03-18T20:24:45.430298+0000","last_clean":"2024-03-18T20:24:45.430298+0000","last_became_active":"2024-03-18T20:24:40.940285+0000","last_became_peered":"2024-03-18T20:24:40.940285+0000","last_unstale":"2024-03-18T20:24:45.430298+0000","last_undegraded":"2024-03-18T20:24:45.430298+0000","last_fullsized":"2024-03-18T20:24:45.430298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:03.933412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442035+0000","last_change":"2024-03-18T20:24:41.370212+0000","last_active":"2024-03-18T20:24:45.442035+0000","last_peered":"2024-03-18T20:24:45.442035+0000","last_clean":"2024-03-18T20:24:45.442035+0000","last_became_active":"2024-03-18T20:24:41.369904+0000","last_became_peered":"2024-03-18T20:24:41.369904+0000","last_unstale":"2024-03-18T20:24:45.442035+0000","last_undegraded":"2024-03-18T20:24:45.442035+0000","last_fullsized":"2024-03-18T20:24:45.442035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:13:12.982929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961796+0000","last_change":"2024-03-18T20:24:45.446935+0000","last_active":"2024-03-18T20:24:45.961796+0000","last_peered":"2024-03-18T20:24:45.961796+0000","last_clean":"2024-03-18T20:24:45.961796+0000","last_became_active":"2024-03-18T20:24:45.445971+0000","last_became_peered":"2024-03-18T20:24:45.445971+0000","last_unstale":"2024-03-18T20:24:45.961796+0000","last_undegraded":"2024-03-18T20:24:45.961796+0000","last_fullsized":"2024-03-18T20:24:45.961796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:51.292743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470479+0000","last_change":"2024-03-18T20:24:40.937625+0000","last_active":"2024-03-18T20:24:58.470479+0000","last_peered":"2024-03-18T20:24:58.470479+0000","last_clean":"2024-03-18T20:24:58.470479+0000","last_became_active":"2024-03-18T20:24:40.937275+0000","last_became_peered":"2024-03-18T20:24:40.937275+0000","last_unstale":"2024-03-18T20:24:58.470479+0000","last_undegraded":"2024-03-18T20:24:58.470479+0000","last_fullsized":"2024-03-18T20:24:58.470479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:55.747307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430187+0000","last_change":"2024-03-18T20:24:41.349528+0000","last_active":"2024-03-18T20:24:45.430187+0000","last_peered":"2024-03-18T20:24:45.430187+0000","last_clean":"2024-03-18T20:24:45.430187+0000","last_became_active":"2024-03-18T20:24:41.349113+0000","last_became_peered":"2024-03-18T20:24:41.349113+0000","last_unstale":"2024-03-18T20:24:45.430187+0000","last_undegraded":"2024-03-18T20:24:45.430187+0000","last_fullsized":"2024-03-18T20:24:45.430187+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:06.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444468+0000","last_change":"2024-03-18T20:24:45.441827+0000","last_active":"2024-03-18T20:24:45.444468+0000","last_peered":"2024-03-18T20:24:45.444468+0000","last_clean":"2024-03-18T20:24:45.444468+0000","last_became_active":"2024-03-18T20:24:45.441403+0000","last_became_peered":"2024-03-18T20:24:45.441403+0000","last_unstale":"2024-03-18T20:24:45.444468+0000","last_undegraded":"2024-03-18T20:24:45.444468+0000","last_fullsized":"2024-03-18T20:24:45.444468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:42.530843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470136+0000","last_change":"2024-03-18T20:24:40.943702+0000","last_active":"2024-03-18T20:24:58.470136+0000","last_peered":"2024-03-18T20:24:58.470136+0000","last_clean":"2024-03-18T20:24:58.470136+0000","last_became_active":"2024-03-18T20:24:40.943380+0000","last_became_peered":"2024-03-18T20:24:40.943380+0000","last_unstale":"2024-03-18T20:24:58.470136+0000","last_undegraded":"2024-03-18T20:24:58.470136+0000","last_fullsized":"2024-03-18T20:24:58.470136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:04.567157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431139+0000","last_change":"2024-03-18T20:24:41.352181+0000","last_active":"2024-03-18T20:24:45.431139+0000","last_peered":"2024-03-18T20:24:45.431139+0000","last_clean":"2024-03-18T20:24:45.431139+0000","last_became_active":"2024-03-18T20:24:41.351694+0000","last_became_peered":"2024-03-18T20:24:41.351694+0000","last_unstale":"2024-03-18T20:24:45.431139+0000","last_undegraded":"2024-03-18T20:24:45.431139+0000","last_fullsized":"2024-03-18T20:24:45.431139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:00.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961823+0000","last_change":"2024-03-18T20:24:45.444567+0000","last_active":"2024-03-18T20:24:45.961823+0000","last_peered":"2024-03-18T20:24:45.961823+0000","last_clean":"2024-03-18T20:24:45.961823+0000","last_became_active":"2024-03-18T20:24:45.443996+0000","last_became_peered":"2024-03-18T20:24:45.443996+0000","last_unstale":"2024-03-18T20:24:45.961823+0000","last_undegraded":"2024-03-18T20:24:45.961823+0000","last_fullsized":"2024-03-18T20:24:45.961823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:16.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426427+0000","last_change":"2024-03-18T20:24:40.948872+0000","last_active":"2024-03-18T20:24:45.426427+0000","last_peered":"2024-03-18T20:24:45.426427+0000","last_clean":"2024-03-18T20:24:45.426427+0000","last_became_active":"2024-03-18T20:24:40.948540+0000","last_became_peered":"2024-03-18T20:24:40.948540+0000","last_unstale":"2024-03-18T20:24:45.426427+0000","last_undegraded":"2024-03-18T20:24:45.426427+0000","last_fullsized":"2024-03-18T20:24:45.426427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:57.954340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426231+0000","last_change":"2024-03-18T20:24:41.355552+0000","last_active":"2024-03-18T20:24:45.426231+0000","last_peered":"2024-03-18T20:24:45.426231+0000","last_clean":"2024-03-18T20:24:45.426231+0000","last_became_active":"2024-03-18T20:24:41.355245+0000","last_became_peered":"2024-03-18T20:24:41.355245+0000","last_unstale":"2024-03-18T20:24:45.426231+0000","last_undegraded":"2024-03-18T20:24:45.426231+0000","last_fullsized":"2024-03-18T20:24:45.426231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:25.833815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961749+0000","last_change":"2024-03-18T20:24:45.443078+0000","last_active":"2024-03-18T20:24:45.961749+0000","last_peered":"2024-03-18T20:24:45.961749+0000","last_clean":"2024-03-18T20:24:45.961749+0000","last_became_active":"2024-03-18T20:24:45.442675+0000","last_became_peered":"2024-03-18T20:24:45.442675+0000","last_unstale":"2024-03-18T20:24:45.961749+0000","last_undegraded":"2024-03-18T20:24:45.961749+0000","last_fullsized":"2024-03-18T20:24:45.961749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:32.262956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472041+0000","last_change":"2024-03-18T20:24:40.943239+0000","last_active":"2024-03-18T20:24:58.472041+0000","last_peered":"2024-03-18T20:24:58.472041+0000","last_clean":"2024-03-18T20:24:58.472041+0000","last_became_active":"2024-03-18T20:24:40.942862+0000","last_became_peered":"2024-03-18T20:24:40.942862+0000","last_unstale":"2024-03-18T20:24:58.472041+0000","last_undegraded":"2024-03-18T20:24:58.472041+0000","last_fullsized":"2024-03-18T20:24:58.472041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:42.560340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437303+0000","last_change":"2024-03-18T20:24:41.361986+0000","last_active":"2024-03-18T20:24:45.437303+0000","last_peered":"2024-03-18T20:24:45.437303+0000","last_clean":"2024-03-18T20:24:45.437303+0000","last_became_active":"2024-03-18T20:24:41.361608+0000","last_became_peered":"2024-03-18T20:24:41.361608+0000","last_unstale":"2024-03-18T20:24:45.437303+0000","last_undegraded":"2024-03-18T20:24:45.437303+0000","last_fullsized":"2024-03-18T20:24:45.437303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:44.540156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441527+0000","last_change":"2024-03-18T20:24:45.440250+0000","last_active":"2024-03-18T20:24:45.441527+0000","last_peered":"2024-03-18T20:24:45.441527+0000","last_clean":"2024-03-18T20:24:45.441527+0000","last_became_active":"2024-03-18T20:24:45.439782+0000","last_became_peered":"2024-03-18T20:24:45.439782+0000","last_unstale":"2024-03-18T20:24:45.441527+0000","last_undegraded":"2024-03-18T20:24:45.441527+0000","last_fullsized":"2024-03-18T20:24:45.441527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:58.415354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427038+0000","last_change":"2024-03-18T20:24:40.945951+0000","last_active":"2024-03-18T20:24:45.427038+0000","last_peered":"2024-03-18T20:24:45.427038+0000","last_clean":"2024-03-18T20:24:45.427038+0000","last_became_active":"2024-03-18T20:24:40.945147+0000","last_became_peered":"2024-03-18T20:24:40.945147+0000","last_unstale":"2024-03-18T20:24:45.427038+0000","last_undegraded":"2024-03-18T20:24:45.427038+0000","last_fullsized":"2024-03-18T20:24:45.427038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:45.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436691+0000","last_change":"2024-03-18T20:24:41.366752+0000","last_active":"2024-03-18T20:24:45.436691+0000","last_peered":"2024-03-18T20:24:45.436691+0000","last_clean":"2024-03-18T20:24:45.436691+0000","last_became_active":"2024-03-18T20:24:41.365949+0000","last_became_peered":"2024-03-18T20:24:41.365949+0000","last_unstale":"2024-03-18T20:24:45.436691+0000","last_undegraded":"2024-03-18T20:24:45.436691+0000","last_fullsized":"2024-03-18T20:24:45.436691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:42.785307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962639+0000","last_change":"2024-03-18T20:24:45.446303+0000","last_active":"2024-03-18T20:24:45.962639+0000","last_peered":"2024-03-18T20:24:45.962639+0000","last_clean":"2024-03-18T20:24:45.962639+0000","last_became_active":"2024-03-18T20:24:45.445873+0000","last_became_peered":"2024-03-18T20:24:45.445873+0000","last_unstale":"2024-03-18T20:24:45.962639+0000","last_undegraded":"2024-03-18T20:24:45.962639+0000","last_fullsized":"2024-03-18T20:24:45.962639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:41.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438335+0000","last_change":"2024-03-18T20:24:40.945490+0000","last_active":"2024-03-18T20:24:45.438335+0000","last_peered":"2024-03-18T20:24:45.438335+0000","last_clean":"2024-03-18T20:24:45.438335+0000","last_became_active":"2024-03-18T20:24:40.945155+0000","last_became_peered":"2024-03-18T20:24:40.945155+0000","last_unstale":"2024-03-18T20:24:45.438335+0000","last_undegraded":"2024-03-18T20:24:45.438335+0000","last_fullsized":"2024-03-18T20:24:45.438335+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:50.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436214+0000","last_change":"2024-03-18T20:24:41.369354+0000","last_active":"2024-03-18T20:24:45.436214+0000","last_peered":"2024-03-18T20:24:45.436214+0000","last_clean":"2024-03-18T20:24:45.436214+0000","last_became_active":"2024-03-18T20:24:41.369079+0000","last_became_peered":"2024-03-18T20:24:41.369079+0000","last_unstale":"2024-03-18T20:24:45.436214+0000","last_undegraded":"2024-03-18T20:24:45.436214+0000","last_fullsized":"2024-03-18T20:24:45.436214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:25.264037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445168+0000","last_change":"2024-03-18T20:24:45.442906+0000","last_active":"2024-03-18T20:24:45.445168+0000","last_peered":"2024-03-18T20:24:45.445168+0000","last_clean":"2024-03-18T20:24:45.445168+0000","last_became_active":"2024-03-18T20:24:45.442318+0000","last_became_peered":"2024-03-18T20:24:45.442318+0000","last_unstale":"2024-03-18T20:24:45.445168+0000","last_undegraded":"2024-03-18T20:24:45.445168+0000","last_fullsized":"2024-03-18T20:24:45.445168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:50.656133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430336+0000","last_change":"2024-03-18T20:24:40.940614+0000","last_active":"2024-03-18T20:24:45.430336+0000","last_peered":"2024-03-18T20:24:45.430336+0000","last_clean":"2024-03-18T20:24:45.430336+0000","last_became_active":"2024-03-18T20:24:40.940089+0000","last_became_peered":"2024-03-18T20:24:40.940089+0000","last_unstale":"2024-03-18T20:24:45.430336+0000","last_undegraded":"2024-03-18T20:24:45.430336+0000","last_fullsized":"2024-03-18T20:24:45.430336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:09.264556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436344+0000","last_change":"2024-03-18T20:24:41.364100+0000","last_active":"2024-03-18T20:24:45.436344+0000","last_peered":"2024-03-18T20:24:45.436344+0000","last_clean":"2024-03-18T20:24:45.436344+0000","last_became_active":"2024-03-18T20:24:41.363696+0000","last_became_peered":"2024-03-18T20:24:41.363696+0000","last_unstale":"2024-03-18T20:24:45.436344+0000","last_undegraded":"2024-03-18T20:24:45.436344+0000","last_fullsized":"2024-03-18T20:24:45.436344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:46.070917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431227+0000","last_change":"2024-03-18T20:24:41.368853+0000","last_active":"2024-03-18T20:24:45.431227+0000","last_peered":"2024-03-18T20:24:45.431227+0000","last_clean":"2024-03-18T20:24:45.431227+0000","last_became_active":"2024-03-18T20:24:41.368568+0000","last_became_peered":"2024-03-18T20:24:41.368568+0000","last_unstale":"2024-03-18T20:24:45.431227+0000","last_undegraded":"2024-03-18T20:24:45.431227+0000","last_fullsized":"2024-03-18T20:24:45.431227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:24.063361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426949+0000","last_change":"2024-03-18T20:24:41.369055+0000","last_active":"2024-03-18T20:24:45.426949+0000","last_peered":"2024-03-18T20:24:45.426949+0000","last_clean":"2024-03-18T20:24:45.426949+0000","last_became_active":"2024-03-18T20:24:41.368726+0000","last_became_peered":"2024-03-18T20:24:41.368726+0000","last_unstale":"2024-03-18T20:24:45.426949+0000","last_undegraded":"2024-03-18T20:24:45.426949+0000","last_fullsized":"2024-03-18T20:24:45.426949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:15.454756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426328+0000","last_change":"2024-03-18T20:24:41.358811+0000","last_active":"2024-03-18T20:24:45.426328+0000","last_peered":"2024-03-18T20:24:45.426328+0000","last_clean":"2024-03-18T20:24:45.426328+0000","last_became_active":"2024-03-18T20:24:41.357776+0000","last_became_peered":"2024-03-18T20:24:41.357776+0000","last_unstale":"2024-03-18T20:24:45.426328+0000","last_undegraded":"2024-03-18T20:24:45.426328+0000","last_fullsized":"2024-03-18T20:24:45.426328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.069338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442460+0000","last_change":"2024-03-18T20:24:41.370288+0000","last_active":"2024-03-18T20:24:45.442460+0000","last_peered":"2024-03-18T20:24:45.442460+0000","last_clean":"2024-03-18T20:24:45.442460+0000","last_became_active":"2024-03-18T20:24:41.370008+0000","last_became_peered":"2024-03-18T20:24:41.370008+0000","last_unstale":"2024-03-18T20:24:45.442460+0000","last_undegraded":"2024-03-18T20:24:45.442460+0000","last_fullsized":"2024-03-18T20:24:45.442460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:18.246575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430945+0000","last_change":"2024-03-18T20:24:41.370168+0000","last_active":"2024-03-18T20:24:45.430945+0000","last_peered":"2024-03-18T20:24:45.430945+0000","last_clean":"2024-03-18T20:24:45.430945+0000","last_became_active":"2024-03-18T20:24:41.369879+0000","last_became_peered":"2024-03-18T20:24:41.369879+0000","last_unstale":"2024-03-18T20:24:45.430945+0000","last_undegraded":"2024-03-18T20:24:45.430945+0000","last_fullsized":"2024-03-18T20:24:45.430945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:30:42.280869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430414+0000","last_change":"2024-03-18T20:24:41.369036+0000","last_active":"2024-03-18T20:24:45.430414+0000","last_peered":"2024-03-18T20:24:45.430414+0000","last_clean":"2024-03-18T20:24:45.430414+0000","last_became_active":"2024-03-18T20:24:41.368722+0000","last_became_peered":"2024-03-18T20:24:41.368722+0000","last_unstale":"2024-03-18T20:24:45.430414+0000","last_undegraded":"2024-03-18T20:24:45.430414+0000","last_fullsized":"2024-03-18T20:24:45.430414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:11.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427136+0000","last_change":"2024-03-18T20:24:41.355806+0000","last_active":"2024-03-18T20:24:45.427136+0000","last_peered":"2024-03-18T20:24:45.427136+0000","last_clean":"2024-03-18T20:24:45.427136+0000","last_became_active":"2024-03-18T20:24:41.354812+0000","last_became_peered":"2024-03-18T20:24:41.354812+0000","last_unstale":"2024-03-18T20:24:45.427136+0000","last_undegraded":"2024-03-18T20:24:45.427136+0000","last_fullsized":"2024-03-18T20:24:45.427136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:49.307113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426500+0000","last_change":"2024-03-18T20:24:41.371292+0000","last_active":"2024-03-18T20:24:45.426500+0000","last_peered":"2024-03-18T20:24:45.426500+0000","last_clean":"2024-03-18T20:24:45.426500+0000","last_became_active":"2024-03-18T20:24:41.370971+0000","last_became_peered":"2024-03-18T20:24:41.370971+0000","last_unstale":"2024-03-18T20:24:45.426500+0000","last_undegraded":"2024-03-18T20:24:45.426500+0000","last_fullsized":"2024-03-18T20:24:45.426500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:47.568840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438068+0000","last_change":"2024-03-18T20:24:41.368000+0000","last_active":"2024-03-18T20:24:45.438068+0000","last_peered":"2024-03-18T20:24:45.438068+0000","last_clean":"2024-03-18T20:24:45.438068+0000","last_became_active":"2024-03-18T20:24:41.367671+0000","last_became_peered":"2024-03-18T20:24:41.367671+0000","last_unstale":"2024-03-18T20:24:45.438068+0000","last_undegraded":"2024-03-18T20:24:45.438068+0000","last_fullsized":"2024-03-18T20:24:45.438068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:17.940008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425913+0000","last_change":"2024-03-18T20:24:41.354675+0000","last_active":"2024-03-18T20:24:45.425913+0000","last_peered":"2024-03-18T20:24:45.425913+0000","last_clean":"2024-03-18T20:24:45.425913+0000","last_became_active":"2024-03-18T20:24:41.354358+0000","last_became_peered":"2024-03-18T20:24:41.354358+0000","last_unstale":"2024-03-18T20:24:45.425913+0000","last_undegraded":"2024-03-18T20:24:45.425913+0000","last_fullsized":"2024-03-18T20:24:45.425913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:47.238810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433664+0000","last_change":"2024-03-18T20:24:41.360057+0000","last_active":"2024-03-18T20:24:45.433664+0000","last_peered":"2024-03-18T20:24:45.433664+0000","last_clean":"2024-03-18T20:24:45.433664+0000","last_became_active":"2024-03-18T20:24:41.359704+0000","last_became_peered":"2024-03-18T20:24:41.359704+0000","last_unstale":"2024-03-18T20:24:45.433664+0000","last_undegraded":"2024-03-18T20:24:45.433664+0000","last_fullsized":"2024-03-18T20:24:45.433664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:22.235537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430704+0000","last_change":"2024-03-18T20:24:41.357847+0000","last_active":"2024-03-18T20:24:45.430704+0000","last_peered":"2024-03-18T20:24:45.430704+0000","last_clean":"2024-03-18T20:24:45.430704+0000","last_became_active":"2024-03-18T20:24:41.357555+0000","last_became_peered":"2024-03-18T20:24:41.357555+0000","last_unstale":"2024-03-18T20:24:45.430704+0000","last_undegraded":"2024-03-18T20:24:45.430704+0000","last_fullsized":"2024-03-18T20:24:45.430704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:01.049713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431110+0000","last_change":"2024-03-18T20:24:41.366158+0000","last_active":"2024-03-18T20:24:45.431110+0000","last_peered":"2024-03-18T20:24:45.431110+0000","last_clean":"2024-03-18T20:24:45.431110+0000","last_became_active":"2024-03-18T20:24:41.365810+0000","last_became_peered":"2024-03-18T20:24:41.365810+0000","last_unstale":"2024-03-18T20:24:45.431110+0000","last_undegraded":"2024-03-18T20:24:45.431110+0000","last_fullsized":"2024-03-18T20:24:45.431110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:46:39.376215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6063,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":103,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":19181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514508,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":457703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776144,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1053115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.94299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.75800000000000001}]},{"osd":9,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.044},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.73499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":9,"up_from":63,"seq":270582939668,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":458521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.376},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.33900000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.443}]},{"osd":3,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.58199999999999996}]},{"osd":8,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59999999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.627},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":8,"up_from":56,"seq":240518168599,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":458110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.441},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68200000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78300000000000003}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":7,"up_from":49,"seq":210453397531,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1049767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.499}]},{"osd":2,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.436}]},{"osd":5,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.628},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51900000000000002}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67500000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68899999999999995}]},{"osd":10,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561053,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":457703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.372,"15min":0.36899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.308},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.27400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.378}]},{"osd":4,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.439}]},{"osd":5,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.67900000000000005,"15min":0.63300000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":5.508,"5min":5.508,"15min":5.508},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.40899999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.55600000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.496}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":461225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.46899999999999997},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.495,"15min":0.47999999999999998},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66300000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71399999999999997,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.438}]},{"osd":3,"last update":"Mon Mar 18 20:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.42999999999999999},"min":{"1min":0.32000000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.878},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.72999999999999998}]},{"osd":8,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74399999999999999}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.755}]},{"osd":10,"last update":"Mon Mar 18 20:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.88300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":457793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.48999999999999999},"min":{"1min":0.33900000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.49399999999999999},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.53200000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.5,"15min":0.48799999999999999},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53300000000000003,"15min":0.499},"min":{"1min":0.45700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.32800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.505}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76700000000000002,"5min":1.165,"15min":1.165},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.501,"15min":0.505},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.67300000000000004}]},{"osd":8,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56000000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.496},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.621}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.57799999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":461259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78700000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.80300000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63400000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:24:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.33300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.81299999999999994}]},{"osd":8,"last update":"Mon Mar 18 20:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.62}]},{"osd":9,"last update":"Mon Mar 18 20:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.60699999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.749},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.77400000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":457530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33100000000000002},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.255,"5min":0.221,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.745,"5min":1.256,"15min":1.256},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83399999999999996,"5min":1.228,"15min":1.228},"last":0.63},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":1.224,"15min":1.224},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.621}]},{"osd":5,"last update":"Mon Mar 18 20:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79700000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.498}]},{"osd":7,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.54500000000000004}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.67000000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55500000000000005}]},{"osd":10,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.452,"5min":7.452,"15min":7.452},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.72199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":937984,"data_stored":457725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.49199999999999999,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51000000000000001,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.442}]},{"osd":1,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.32300000000000001},"min":{"1min":0.255,"5min":0.254,"15min":0.254},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.29699999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.496,"15min":0.48099999999999998},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.622},{"interface":"front","average":{"1min":0.628,"5min":0.504,"15min":0.48299999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.58399999999999996,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.96199999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.89200000000000002}]},{"osd":9,"last update":"Mon Mar 18 20:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.83599999999999997}]},{"osd":10,"last update":"Mon Mar 18 20:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.92700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789986,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1049499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.751,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.42199999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.255,"5min":0.245,"15min":0.245},"max":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.432,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.41499999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79000000000000004,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.879,"15min":0.879},"last":0.52000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.62}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.504},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67800000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.68600000000000005}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:03.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:03.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 -- 172.21.15.98:0/800883866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f81016f0 msgr2=0x7f44f8103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/800883866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f81016f0 0x7f44f8103ae0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009990 tx=0x7f44e8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 -- 172.21.15.98:0/800883866 shutdown_connections 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/800883866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8106950 0x7f44f8108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/800883866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f81016f0 0x7f44f8103ae0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/800883866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f8104020 0x7f44f8106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 -- 172.21.15.98:0/800883866 >> 172.21.15.98:0/800883866 conn(0x7f44f80fb0e0 msgr2=0x7f44f80fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 -- 172.21.15.98:0/800883866 shutdown_connections 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.749+0000 7f44fd6dd700 1 -- 172.21.15.98:0/800883866 wait complete. 2024-03-18T20:25:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.750+0000 7f44fd6dd700 1 Processor -- start 2024-03-18T20:25:03.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.750+0000 7f44fd6dd700 1 -- start start 2024-03-18T20:25:03.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.750+0000 7f44fd6dd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f81016f0 0x7f44f8196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.750+0000 7f44fd6dd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8104020 0x7f44f8197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f819e630 con 0x7f44f8106950 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f819e770 con 0x7f44f81016f0 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f819e900 con 0x7f44f8104020 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32768/0 (socket says 172.21.15.98:32768) 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 -- 172.21.15.98:0/640111780 learned_addr learned my addr 172.21.15.98:0/640111780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f6ffd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f81016f0 0x7f44f8196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f67fc700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8104020 0x7f44f8197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 -- 172.21.15.98:0/640111780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8104020 msgr2=0x7f44f8197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8104020 0x7f44f8197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 -- 172.21.15.98:0/640111780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f81016f0 msgr2=0x7f44f8196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f81016f0 0x7f44f8196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 -- 172.21.15.98:0/640111780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80095f0 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44f77fe700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b970 tx=0x7f44f000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f000d860 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f44f000de50 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f819ebe0 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f0017400 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.751+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f819f0a0 con 0x7f44f8106950 2024-03-18T20:25:03.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.752+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f80fcc60 con 0x7f44f8106950 2024-03-18T20:25:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.753+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f44f000d9c0 con 0x7f44f8106950 2024-03-18T20:25:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.754+0000 7f44dffff700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 0x7f44e00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.754+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f44f0006e40 con 0x7f44f8106950 2024-03-18T20:25:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.754+0000 7f44f6ffd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 0x7f44e00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.754+0000 7f44f6ffd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 0x7f44e00a3670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.758+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f44f0095ce0 con 0x7f44f8106950 2024-03-18T20:25:03.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:03 smithi098 ceph-mon[43462]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:03 smithi098 ceph-mon[43462]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3053116688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:03.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3514191263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:03.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:03 smithi169 ceph-mon[48529]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:03 smithi169 ceph-mon[48529]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:03 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3053116688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:03.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:03 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3514191263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:03.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.949+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44f80611f0 con 0x7f44e00a11b0 2024-03-18T20:25:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.961+0000 7f44dffff700 1 -- 172.21.15.98:0/640111780 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+417438 (secure 0 0 0) 0x7f44f80611f0 con 0x7f44e00a11b0 2024-03-18T20:25:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 msgr2=0x7f44e00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 0x7f44e00a3670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 msgr2=0x7f44f819e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f44f000b970 tx=0x7f44f000bc80 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 shutdown_connections 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44f8106950 0x7f44f819e060 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f44e00a11b0 0x7f44e00a3670 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44f81016f0 0x7f44f8196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 --2- 172.21.15.98:0/640111780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44f8104020 0x7f44f8197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 >> 172.21.15.98:0/640111780 conn(0x7f44f80fb0e0 msgr2=0x7f44f8102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 shutdown_connections 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:03.965+0000 7f44fd6dd700 1 -- 172.21.15.98:0/640111780 wait complete. 2024-03-18T20:25:03.976 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-18T20:25:03.981 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-03-18T20:25:03.575649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597604,"num_objects":58,"num_object_clones":0,"num_object_copies":175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":148,"num_read_kb":130,"num_write":302,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337068,"kb_used_data":12876,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604756,"statfs":{"total":1151940427776,"available":1151595270144,"internally_reserved":0,"allocated":13185024,"data_stored":7282394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4874,"num_objects":34,"num_object_clones":0,"num_object_copies":103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":22,"num_read_kb":21,"num_write":76,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430988+0000","last_change":"2024-03-18T20:24:41.360546+0000","last_active":"2024-03-18T20:24:45.430988+0000","last_peered":"2024-03-18T20:24:45.430988+0000","last_clean":"2024-03-18T20:24:45.430988+0000","last_became_active":"2024-03-18T20:24:41.360221+0000","last_became_peered":"2024-03-18T20:24:41.360221+0000","last_unstale":"2024-03-18T20:24:45.430988+0000","last_undegraded":"2024-03-18T20:24:45.430988+0000","last_fullsized":"2024-03-18T20:24:45.430988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:23.329362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441941+0000","last_change":"2024-03-18T20:24:41.354655+0000","last_active":"2024-03-18T20:24:45.441941+0000","last_peered":"2024-03-18T20:24:45.441941+0000","last_clean":"2024-03-18T20:24:45.441941+0000","last_became_active":"2024-03-18T20:24:41.354340+0000","last_became_peered":"2024-03-18T20:24:41.354340+0000","last_unstale":"2024-03-18T20:24:45.441941+0000","last_undegraded":"2024-03-18T20:24:45.441941+0000","last_fullsized":"2024-03-18T20:24:45.441941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:18.785662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440207+0000","last_change":"2024-03-18T20:24:41.351670+0000","last_active":"2024-03-18T20:24:45.440207+0000","last_peered":"2024-03-18T20:24:45.440207+0000","last_clean":"2024-03-18T20:24:45.440207+0000","last_became_active":"2024-03-18T20:24:41.351379+0000","last_became_peered":"2024-03-18T20:24:41.351379+0000","last_unstale":"2024-03-18T20:24:45.440207+0000","last_undegraded":"2024-03-18T20:24:45.440207+0000","last_fullsized":"2024-03-18T20:24:45.440207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:50.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436769+0000","last_change":"2024-03-18T20:24:41.364821+0000","last_active":"2024-03-18T20:24:45.436769+0000","last_peered":"2024-03-18T20:24:45.436769+0000","last_clean":"2024-03-18T20:24:45.436769+0000","last_became_active":"2024-03-18T20:24:41.364496+0000","last_became_peered":"2024-03-18T20:24:41.364496+0000","last_unstale":"2024-03-18T20:24:45.436769+0000","last_undegraded":"2024-03-18T20:24:45.436769+0000","last_fullsized":"2024-03-18T20:24:45.436769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:53.506013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431016+0000","last_change":"2024-03-18T20:24:41.356223+0000","last_active":"2024-03-18T20:24:45.431016+0000","last_peered":"2024-03-18T20:24:45.431016+0000","last_clean":"2024-03-18T20:24:45.431016+0000","last_became_active":"2024-03-18T20:24:41.355831+0000","last_became_peered":"2024-03-18T20:24:41.355831+0000","last_unstale":"2024-03-18T20:24:45.431016+0000","last_undegraded":"2024-03-18T20:24:45.431016+0000","last_fullsized":"2024-03-18T20:24:45.431016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:07.593472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852430+0000","last_change":"2024-03-18T20:24:41.367208+0000","last_active":"2024-03-18T20:24:45.852430+0000","last_peered":"2024-03-18T20:24:45.852430+0000","last_clean":"2024-03-18T20:24:45.852430+0000","last_became_active":"2024-03-18T20:24:41.366885+0000","last_became_peered":"2024-03-18T20:24:41.366885+0000","last_unstale":"2024-03-18T20:24:45.852430+0000","last_undegraded":"2024-03-18T20:24:45.852430+0000","last_fullsized":"2024-03-18T20:24:45.852430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:34.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436116+0000","last_change":"2024-03-18T20:24:41.367935+0000","last_active":"2024-03-18T20:24:45.436116+0000","last_peered":"2024-03-18T20:24:45.436116+0000","last_clean":"2024-03-18T20:24:45.436116+0000","last_became_active":"2024-03-18T20:24:41.367615+0000","last_became_peered":"2024-03-18T20:24:41.367615+0000","last_unstale":"2024-03-18T20:24:45.436116+0000","last_undegraded":"2024-03-18T20:24:45.436116+0000","last_fullsized":"2024-03-18T20:24:45.436116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:41.242857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426070+0000","last_change":"2024-03-18T20:24:41.369045+0000","last_active":"2024-03-18T20:24:45.426070+0000","last_peered":"2024-03-18T20:24:45.426070+0000","last_clean":"2024-03-18T20:24:45.426070+0000","last_became_active":"2024-03-18T20:24:41.368669+0000","last_became_peered":"2024-03-18T20:24:41.368669+0000","last_unstale":"2024-03-18T20:24:45.426070+0000","last_undegraded":"2024-03-18T20:24:45.426070+0000","last_fullsized":"2024-03-18T20:24:45.426070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:09.538243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432055+0000","last_change":"2024-03-18T20:24:41.366520+0000","last_active":"2024-03-18T20:24:45.432055+0000","last_peered":"2024-03-18T20:24:45.432055+0000","last_clean":"2024-03-18T20:24:45.432055+0000","last_became_active":"2024-03-18T20:24:41.366155+0000","last_became_peered":"2024-03-18T20:24:41.366155+0000","last_unstale":"2024-03-18T20:24:45.432055+0000","last_undegraded":"2024-03-18T20:24:45.432055+0000","last_fullsized":"2024-03-18T20:24:45.432055+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:33.151394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852478+0000","last_change":"2024-03-18T20:24:41.370742+0000","last_active":"2024-03-18T20:24:45.852478+0000","last_peered":"2024-03-18T20:24:45.852478+0000","last_clean":"2024-03-18T20:24:45.852478+0000","last_became_active":"2024-03-18T20:24:41.370438+0000","last_became_peered":"2024-03-18T20:24:41.370438+0000","last_unstale":"2024-03-18T20:24:45.852478+0000","last_undegraded":"2024-03-18T20:24:45.852478+0000","last_fullsized":"2024-03-18T20:24:45.852478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:04.331608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436604+0000","last_change":"2024-03-18T20:24:41.366970+0000","last_active":"2024-03-18T20:24:45.436604+0000","last_peered":"2024-03-18T20:24:45.436604+0000","last_clean":"2024-03-18T20:24:45.436604+0000","last_became_active":"2024-03-18T20:24:41.366440+0000","last_became_peered":"2024-03-18T20:24:41.366440+0000","last_unstale":"2024-03-18T20:24:45.436604+0000","last_undegraded":"2024-03-18T20:24:45.436604+0000","last_fullsized":"2024-03-18T20:24:45.436604+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:10.050606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436670+0000","last_change":"2024-03-18T20:24:41.372375+0000","last_active":"2024-03-18T20:24:45.436670+0000","last_peered":"2024-03-18T20:24:45.436670+0000","last_clean":"2024-03-18T20:24:45.436670+0000","last_became_active":"2024-03-18T20:24:41.371028+0000","last_became_peered":"2024-03-18T20:24:41.371028+0000","last_unstale":"2024-03-18T20:24:45.436670+0000","last_undegraded":"2024-03-18T20:24:45.436670+0000","last_fullsized":"2024-03-18T20:24:45.436670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:39.906774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430787+0000","last_change":"2024-03-18T20:24:41.368252+0000","last_active":"2024-03-18T20:24:45.430787+0000","last_peered":"2024-03-18T20:24:45.430787+0000","last_clean":"2024-03-18T20:24:45.430787+0000","last_became_active":"2024-03-18T20:24:41.367911+0000","last_became_peered":"2024-03-18T20:24:41.367911+0000","last_unstale":"2024-03-18T20:24:45.430787+0000","last_undegraded":"2024-03-18T20:24:45.430787+0000","last_fullsized":"2024-03-18T20:24:45.430787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:36.250924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440015+0000","last_change":"2024-03-18T20:24:41.358690+0000","last_active":"2024-03-18T20:24:45.440015+0000","last_peered":"2024-03-18T20:24:45.440015+0000","last_clean":"2024-03-18T20:24:45.440015+0000","last_became_active":"2024-03-18T20:24:41.358254+0000","last_became_peered":"2024-03-18T20:24:41.358254+0000","last_unstale":"2024-03-18T20:24:45.440015+0000","last_undegraded":"2024-03-18T20:24:45.440015+0000","last_fullsized":"2024-03-18T20:24:45.440015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:51.466799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440570+0000","last_change":"2024-03-18T20:24:41.364337+0000","last_active":"2024-03-18T20:24:45.440570+0000","last_peered":"2024-03-18T20:24:45.440570+0000","last_clean":"2024-03-18T20:24:45.440570+0000","last_became_active":"2024-03-18T20:24:41.364026+0000","last_became_peered":"2024-03-18T20:24:41.364026+0000","last_unstale":"2024-03-18T20:24:45.440570+0000","last_undegraded":"2024-03-18T20:24:45.440570+0000","last_fullsized":"2024-03-18T20:24:45.440570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:34.336403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431248+0000","last_change":"2024-03-18T20:24:41.360989+0000","last_active":"2024-03-18T20:24:45.431248+0000","last_peered":"2024-03-18T20:24:45.431248+0000","last_clean":"2024-03-18T20:24:45.431248+0000","last_became_active":"2024-03-18T20:24:41.360097+0000","last_became_peered":"2024-03-18T20:24:41.360097+0000","last_unstale":"2024-03-18T20:24:45.431248+0000","last_undegraded":"2024-03-18T20:24:45.431248+0000","last_fullsized":"2024-03-18T20:24:45.431248+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:18.522013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430879+0000","last_change":"2024-03-18T20:24:41.361221+0000","last_active":"2024-03-18T20:24:45.430879+0000","last_peered":"2024-03-18T20:24:45.430879+0000","last_clean":"2024-03-18T20:24:45.430879+0000","last_became_active":"2024-03-18T20:24:41.360924+0000","last_became_peered":"2024-03-18T20:24:41.360924+0000","last_unstale":"2024-03-18T20:24:45.430879+0000","last_undegraded":"2024-03-18T20:24:45.430879+0000","last_fullsized":"2024-03-18T20:24:45.430879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:24.233761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430265+0000","last_change":"2024-03-18T20:24:41.358145+0000","last_active":"2024-03-18T20:24:45.430265+0000","last_peered":"2024-03-18T20:24:45.430265+0000","last_clean":"2024-03-18T20:24:45.430265+0000","last_became_active":"2024-03-18T20:24:41.357862+0000","last_became_peered":"2024-03-18T20:24:41.357862+0000","last_unstale":"2024-03-18T20:24:45.430265+0000","last_undegraded":"2024-03-18T20:24:45.430265+0000","last_fullsized":"2024-03-18T20:24:45.430265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:01.582221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427416+0000","last_change":"2024-03-18T20:24:41.359214+0000","last_active":"2024-03-18T20:24:45.427416+0000","last_peered":"2024-03-18T20:24:45.427416+0000","last_clean":"2024-03-18T20:24:45.427416+0000","last_became_active":"2024-03-18T20:24:41.357976+0000","last_became_peered":"2024-03-18T20:24:41.357976+0000","last_unstale":"2024-03-18T20:24:45.427416+0000","last_undegraded":"2024-03-18T20:24:45.427416+0000","last_fullsized":"2024-03-18T20:24:45.427416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:36.873383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854870+0000","last_change":"2024-03-18T20:24:45.440427+0000","last_active":"2024-03-18T20:24:45.854870+0000","last_peered":"2024-03-18T20:24:45.854870+0000","last_clean":"2024-03-18T20:24:45.854870+0000","last_became_active":"2024-03-18T20:24:45.440023+0000","last_became_peered":"2024-03-18T20:24:45.440023+0000","last_unstale":"2024-03-18T20:24:45.854870+0000","last_undegraded":"2024-03-18T20:24:45.854870+0000","last_fullsized":"2024-03-18T20:24:45.854870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:37.745854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479306+0000","last_change":"2024-03-18T20:24:40.945829+0000","last_active":"2024-03-18T20:24:59.479306+0000","last_peered":"2024-03-18T20:24:59.479306+0000","last_clean":"2024-03-18T20:24:59.479306+0000","last_became_active":"2024-03-18T20:24:40.945498+0000","last_became_peered":"2024-03-18T20:24:40.945498+0000","last_unstale":"2024-03-18T20:24:59.479306+0000","last_undegraded":"2024-03-18T20:24:59.479306+0000","last_fullsized":"2024-03-18T20:24:59.479306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:40.221678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426446+0000","last_change":"2024-03-18T20:24:41.368911+0000","last_active":"2024-03-18T20:24:45.426446+0000","last_peered":"2024-03-18T20:24:45.426446+0000","last_clean":"2024-03-18T20:24:45.426446+0000","last_became_active":"2024-03-18T20:24:41.368603+0000","last_became_peered":"2024-03-18T20:24:41.368603+0000","last_unstale":"2024-03-18T20:24:45.426446+0000","last_undegraded":"2024-03-18T20:24:45.426446+0000","last_fullsized":"2024-03-18T20:24:45.426446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:37.722935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961438+0000","last_change":"2024-03-18T20:24:45.439904+0000","last_active":"2024-03-18T20:24:45.961438+0000","last_peered":"2024-03-18T20:24:45.961438+0000","last_clean":"2024-03-18T20:24:45.961438+0000","last_became_active":"2024-03-18T20:24:45.439489+0000","last_became_peered":"2024-03-18T20:24:45.439489+0000","last_unstale":"2024-03-18T20:24:45.961438+0000","last_undegraded":"2024-03-18T20:24:45.961438+0000","last_fullsized":"2024-03-18T20:24:45.961438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:21.696148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962086+0000","last_change":"2024-03-18T20:24:40.942150+0000","last_active":"2024-03-18T20:24:45.962086+0000","last_peered":"2024-03-18T20:24:45.962086+0000","last_clean":"2024-03-18T20:24:45.962086+0000","last_became_active":"2024-03-18T20:24:40.941848+0000","last_became_peered":"2024-03-18T20:24:40.941848+0000","last_unstale":"2024-03-18T20:24:45.962086+0000","last_undegraded":"2024-03-18T20:24:45.962086+0000","last_fullsized":"2024-03-18T20:24:45.962086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:44:08.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442127+0000","last_change":"2024-03-18T20:24:41.359112+0000","last_active":"2024-03-18T20:24:45.442127+0000","last_peered":"2024-03-18T20:24:45.442127+0000","last_clean":"2024-03-18T20:24:45.442127+0000","last_became_active":"2024-03-18T20:24:41.358817+0000","last_became_peered":"2024-03-18T20:24:41.358817+0000","last_unstale":"2024-03-18T20:24:45.442127+0000","last_undegraded":"2024-03-18T20:24:45.442127+0000","last_fullsized":"2024-03-18T20:24:45.442127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:25:15.699466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962679+0000","last_change":"2024-03-18T20:24:45.445996+0000","last_active":"2024-03-18T20:24:45.962679+0000","last_peered":"2024-03-18T20:24:45.962679+0000","last_clean":"2024-03-18T20:24:45.962679+0000","last_became_active":"2024-03-18T20:24:45.445552+0000","last_became_peered":"2024-03-18T20:24:45.445552+0000","last_unstale":"2024-03-18T20:24:45.962679+0000","last_undegraded":"2024-03-18T20:24:45.962679+0000","last_fullsized":"2024-03-18T20:24:45.962679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:26.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476209+0000","last_change":"2024-03-18T20:24:40.951499+0000","last_active":"2024-03-18T20:24:59.476209+0000","last_peered":"2024-03-18T20:24:59.476209+0000","last_clean":"2024-03-18T20:24:59.476209+0000","last_became_active":"2024-03-18T20:24:40.951183+0000","last_became_peered":"2024-03-18T20:24:40.951183+0000","last_unstale":"2024-03-18T20:24:59.476209+0000","last_undegraded":"2024-03-18T20:24:59.476209+0000","last_fullsized":"2024-03-18T20:24:59.476209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:25.936370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432254+0000","last_change":"2024-03-18T20:24:41.354835+0000","last_active":"2024-03-18T20:24:45.432254+0000","last_peered":"2024-03-18T20:24:45.432254+0000","last_clean":"2024-03-18T20:24:45.432254+0000","last_became_active":"2024-03-18T20:24:41.354486+0000","last_became_peered":"2024-03-18T20:24:41.354486+0000","last_unstale":"2024-03-18T20:24:45.432254+0000","last_undegraded":"2024-03-18T20:24:45.432254+0000","last_fullsized":"2024-03-18T20:24:45.432254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961610+0000","last_change":"2024-03-18T20:24:45.446772+0000","last_active":"2024-03-18T20:24:45.961610+0000","last_peered":"2024-03-18T20:24:45.961610+0000","last_clean":"2024-03-18T20:24:45.961610+0000","last_became_active":"2024-03-18T20:24:45.446296+0000","last_became_peered":"2024-03-18T20:24:45.446296+0000","last_unstale":"2024-03-18T20:24:45.961610+0000","last_undegraded":"2024-03-18T20:24:45.961610+0000","last_fullsized":"2024-03-18T20:24:45.961610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:41:20.784772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431809+0000","last_change":"2024-03-18T20:24:40.948077+0000","last_active":"2024-03-18T20:24:45.431809+0000","last_peered":"2024-03-18T20:24:45.431809+0000","last_clean":"2024-03-18T20:24:45.431809+0000","last_became_active":"2024-03-18T20:24:40.946970+0000","last_became_peered":"2024-03-18T20:24:40.946970+0000","last_unstale":"2024-03-18T20:24:45.431809+0000","last_undegraded":"2024-03-18T20:24:45.431809+0000","last_fullsized":"2024-03-18T20:24:45.431809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:34.558466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437521+0000","last_change":"2024-03-18T20:24:41.371040+0000","last_active":"2024-03-18T20:24:45.437521+0000","last_peered":"2024-03-18T20:24:45.437521+0000","last_clean":"2024-03-18T20:24:45.437521+0000","last_became_active":"2024-03-18T20:24:41.370654+0000","last_became_peered":"2024-03-18T20:24:41.370654+0000","last_unstale":"2024-03-18T20:24:45.437521+0000","last_undegraded":"2024-03-18T20:24:45.437521+0000","last_fullsized":"2024-03-18T20:24:45.437521+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854817+0000","last_change":"2024-03-18T20:24:45.438825+0000","last_active":"2024-03-18T20:24:45.854817+0000","last_peered":"2024-03-18T20:24:45.854817+0000","last_clean":"2024-03-18T20:24:45.854817+0000","last_became_active":"2024-03-18T20:24:45.438418+0000","last_became_peered":"2024-03-18T20:24:45.438418+0000","last_unstale":"2024-03-18T20:24:45.854817+0000","last_undegraded":"2024-03-18T20:24:45.854817+0000","last_fullsized":"2024-03-18T20:24:45.854817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:52.562628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472147+0000","last_change":"2024-03-18T20:24:40.941503+0000","last_active":"2024-03-18T20:24:58.472147+0000","last_peered":"2024-03-18T20:24:58.472147+0000","last_clean":"2024-03-18T20:24:58.472147+0000","last_became_active":"2024-03-18T20:24:40.941150+0000","last_became_peered":"2024-03-18T20:24:40.941150+0000","last_unstale":"2024-03-18T20:24:58.472147+0000","last_undegraded":"2024-03-18T20:24:58.472147+0000","last_fullsized":"2024-03-18T20:24:58.472147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:02.509071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430556+0000","last_change":"2024-03-18T20:24:41.369688+0000","last_active":"2024-03-18T20:24:45.430556+0000","last_peered":"2024-03-18T20:24:45.430556+0000","last_clean":"2024-03-18T20:24:45.430556+0000","last_became_active":"2024-03-18T20:24:41.369361+0000","last_became_peered":"2024-03-18T20:24:41.369361+0000","last_unstale":"2024-03-18T20:24:45.430556+0000","last_undegraded":"2024-03-18T20:24:45.430556+0000","last_fullsized":"2024-03-18T20:24:45.430556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:23.128929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854843+0000","last_change":"2024-03-18T20:24:45.440139+0000","last_active":"2024-03-18T20:24:45.854843+0000","last_peered":"2024-03-18T20:24:45.854843+0000","last_clean":"2024-03-18T20:24:45.854843+0000","last_became_active":"2024-03-18T20:24:45.439758+0000","last_became_peered":"2024-03-18T20:24:45.439758+0000","last_unstale":"2024-03-18T20:24:45.854843+0000","last_undegraded":"2024-03-18T20:24:45.854843+0000","last_fullsized":"2024-03-18T20:24:45.854843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:57.064268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476530+0000","last_change":"2024-03-18T20:24:40.944915+0000","last_active":"2024-03-18T20:24:59.476530+0000","last_peered":"2024-03-18T20:24:59.476530+0000","last_clean":"2024-03-18T20:24:59.476530+0000","last_became_active":"2024-03-18T20:24:40.944547+0000","last_became_peered":"2024-03-18T20:24:40.944547+0000","last_unstale":"2024-03-18T20:24:59.476530+0000","last_undegraded":"2024-03-18T20:24:59.476530+0000","last_fullsized":"2024-03-18T20:24:59.476530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:36.732318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437751+0000","last_change":"2024-03-18T20:24:41.371113+0000","last_active":"2024-03-18T20:24:45.437751+0000","last_peered":"2024-03-18T20:24:45.437751+0000","last_clean":"2024-03-18T20:24:45.437751+0000","last_became_active":"2024-03-18T20:24:41.370764+0000","last_became_peered":"2024-03-18T20:24:41.370764+0000","last_unstale":"2024-03-18T20:24:45.437751+0000","last_undegraded":"2024-03-18T20:24:45.437751+0000","last_fullsized":"2024-03-18T20:24:45.437751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:51.181585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449570+0000","last_change":"2024-03-18T20:24:45.444798+0000","last_active":"2024-03-18T20:24:45.449570+0000","last_peered":"2024-03-18T20:24:45.449570+0000","last_clean":"2024-03-18T20:24:45.449570+0000","last_became_active":"2024-03-18T20:24:45.444374+0000","last_became_peered":"2024-03-18T20:24:45.444374+0000","last_unstale":"2024-03-18T20:24:45.449570+0000","last_undegraded":"2024-03-18T20:24:45.449570+0000","last_fullsized":"2024-03-18T20:24:45.449570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:54.055384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476781+0000","last_change":"2024-03-18T20:24:40.936710+0000","last_active":"2024-03-18T20:24:59.476781+0000","last_peered":"2024-03-18T20:24:59.476781+0000","last_clean":"2024-03-18T20:24:59.476781+0000","last_became_active":"2024-03-18T20:24:40.936360+0000","last_became_peered":"2024-03-18T20:24:40.936360+0000","last_unstale":"2024-03-18T20:24:59.476781+0000","last_undegraded":"2024-03-18T20:24:59.476781+0000","last_fullsized":"2024-03-18T20:24:59.476781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:08.794520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425954+0000","last_change":"2024-03-18T20:24:41.355109+0000","last_active":"2024-03-18T20:24:45.425954+0000","last_peered":"2024-03-18T20:24:45.425954+0000","last_clean":"2024-03-18T20:24:45.425954+0000","last_became_active":"2024-03-18T20:24:41.354718+0000","last_became_peered":"2024-03-18T20:24:41.354718+0000","last_unstale":"2024-03-18T20:24:45.425954+0000","last_undegraded":"2024-03-18T20:24:45.425954+0000","last_fullsized":"2024-03-18T20:24:45.425954+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:39.147267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963393+0000","last_change":"2024-03-18T20:24:45.451275+0000","last_active":"2024-03-18T20:24:45.963393+0000","last_peered":"2024-03-18T20:24:45.963393+0000","last_clean":"2024-03-18T20:24:45.963393+0000","last_became_active":"2024-03-18T20:24:45.450883+0000","last_became_peered":"2024-03-18T20:24:45.450883+0000","last_unstale":"2024-03-18T20:24:45.963393+0000","last_undegraded":"2024-03-18T20:24:45.963393+0000","last_fullsized":"2024-03-18T20:24:45.963393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:10:41.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961539+0000","last_change":"2024-03-18T20:24:40.939692+0000","last_active":"2024-03-18T20:24:45.961539+0000","last_peered":"2024-03-18T20:24:45.961539+0000","last_clean":"2024-03-18T20:24:45.961539+0000","last_became_active":"2024-03-18T20:24:40.939363+0000","last_became_peered":"2024-03-18T20:24:40.939363+0000","last_unstale":"2024-03-18T20:24:45.961539+0000","last_undegraded":"2024-03-18T20:24:45.961539+0000","last_fullsized":"2024-03-18T20:24:45.961539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:41.445337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430758+0000","last_change":"2024-03-18T20:24:41.357835+0000","last_active":"2024-03-18T20:24:45.430758+0000","last_peered":"2024-03-18T20:24:45.430758+0000","last_clean":"2024-03-18T20:24:45.430758+0000","last_became_active":"2024-03-18T20:24:41.357534+0000","last_became_peered":"2024-03-18T20:24:41.357534+0000","last_unstale":"2024-03-18T20:24:45.430758+0000","last_undegraded":"2024-03-18T20:24:45.430758+0000","last_fullsized":"2024-03-18T20:24:45.430758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:16.662257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440398+0000","last_change":"2024-03-18T20:24:45.438295+0000","last_active":"2024-03-18T20:24:45.440398+0000","last_peered":"2024-03-18T20:24:45.440398+0000","last_clean":"2024-03-18T20:24:45.440398+0000","last_became_active":"2024-03-18T20:24:45.437955+0000","last_became_peered":"2024-03-18T20:24:45.437955+0000","last_unstale":"2024-03-18T20:24:45.440398+0000","last_undegraded":"2024-03-18T20:24:45.440398+0000","last_fullsized":"2024-03-18T20:24:45.440398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:26.435144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472214+0000","last_change":"2024-03-18T20:24:40.945507+0000","last_active":"2024-03-18T20:24:58.472214+0000","last_peered":"2024-03-18T20:24:58.472214+0000","last_clean":"2024-03-18T20:24:58.472214+0000","last_became_active":"2024-03-18T20:24:40.945195+0000","last_became_peered":"2024-03-18T20:24:40.945195+0000","last_unstale":"2024-03-18T20:24:58.472214+0000","last_undegraded":"2024-03-18T20:24:58.472214+0000","last_fullsized":"2024-03-18T20:24:58.472214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:29.173130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431923+0000","last_change":"2024-03-18T20:24:41.356795+0000","last_active":"2024-03-18T20:24:45.431923+0000","last_peered":"2024-03-18T20:24:45.431923+0000","last_clean":"2024-03-18T20:24:45.431923+0000","last_became_active":"2024-03-18T20:24:41.356461+0000","last_became_peered":"2024-03-18T20:24:41.356461+0000","last_unstale":"2024-03-18T20:24:45.431923+0000","last_undegraded":"2024-03-18T20:24:45.431923+0000","last_fullsized":"2024-03-18T20:24:45.431923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:04.230985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444215+0000","last_change":"2024-03-18T20:24:45.439984+0000","last_active":"2024-03-18T20:24:45.444215+0000","last_peered":"2024-03-18T20:24:45.444215+0000","last_clean":"2024-03-18T20:24:45.444215+0000","last_became_active":"2024-03-18T20:24:45.439598+0000","last_became_peered":"2024-03-18T20:24:45.439598+0000","last_unstale":"2024-03-18T20:24:45.444215+0000","last_undegraded":"2024-03-18T20:24:45.444215+0000","last_fullsized":"2024-03-18T20:24:45.444215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:49.481812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469645+0000","last_change":"2024-03-18T20:24:40.943978+0000","last_active":"2024-03-18T20:24:58.469645+0000","last_peered":"2024-03-18T20:24:58.469645+0000","last_clean":"2024-03-18T20:24:58.469645+0000","last_became_active":"2024-03-18T20:24:40.943623+0000","last_became_peered":"2024-03-18T20:24:40.943623+0000","last_unstale":"2024-03-18T20:24:58.469645+0000","last_undegraded":"2024-03-18T20:24:58.469645+0000","last_fullsized":"2024-03-18T20:24:58.469645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:58.995256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442197+0000","last_change":"2024-03-18T20:24:41.365465+0000","last_active":"2024-03-18T20:24:45.442197+0000","last_peered":"2024-03-18T20:24:45.442197+0000","last_clean":"2024-03-18T20:24:45.442197+0000","last_became_active":"2024-03-18T20:24:41.365146+0000","last_became_peered":"2024-03-18T20:24:41.365146+0000","last_unstale":"2024-03-18T20:24:45.442197+0000","last_undegraded":"2024-03-18T20:24:45.442197+0000","last_fullsized":"2024-03-18T20:24:45.442197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:47.903106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962967+0000","last_change":"2024-03-18T20:24:45.440065+0000","last_active":"2024-03-18T20:24:45.962967+0000","last_peered":"2024-03-18T20:24:45.962967+0000","last_clean":"2024-03-18T20:24:45.962967+0000","last_became_active":"2024-03-18T20:24:45.439424+0000","last_became_peered":"2024-03-18T20:24:45.439424+0000","last_unstale":"2024-03-18T20:24:45.962967+0000","last_undegraded":"2024-03-18T20:24:45.962967+0000","last_fullsized":"2024-03-18T20:24:45.962967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:28.350851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961988+0000","last_change":"2024-03-18T20:24:40.942424+0000","last_active":"2024-03-18T20:24:45.961988+0000","last_peered":"2024-03-18T20:24:45.961988+0000","last_clean":"2024-03-18T20:24:45.961988+0000","last_became_active":"2024-03-18T20:24:40.941878+0000","last_became_peered":"2024-03-18T20:24:40.941878+0000","last_unstale":"2024-03-18T20:24:45.961988+0000","last_undegraded":"2024-03-18T20:24:45.961988+0000","last_fullsized":"2024-03-18T20:24:45.961988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:40.746350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432427+0000","last_change":"2024-03-18T20:24:41.372684+0000","last_active":"2024-03-18T20:24:45.432427+0000","last_peered":"2024-03-18T20:24:45.432427+0000","last_clean":"2024-03-18T20:24:45.432427+0000","last_became_active":"2024-03-18T20:24:41.372171+0000","last_became_peered":"2024-03-18T20:24:41.372171+0000","last_unstale":"2024-03-18T20:24:45.432427+0000","last_undegraded":"2024-03-18T20:24:45.432427+0000","last_fullsized":"2024-03-18T20:24:45.432427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:50.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444306+0000","last_change":"2024-03-18T20:24:45.440289+0000","last_active":"2024-03-18T20:24:45.444306+0000","last_peered":"2024-03-18T20:24:45.444306+0000","last_clean":"2024-03-18T20:24:45.444306+0000","last_became_active":"2024-03-18T20:24:45.439910+0000","last_became_peered":"2024-03-18T20:24:45.439910+0000","last_unstale":"2024-03-18T20:24:45.444306+0000","last_undegraded":"2024-03-18T20:24:45.444306+0000","last_fullsized":"2024-03-18T20:24:45.444306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:19.420493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.487757+0000","last_change":"2024-03-18T20:24:40.943843+0000","last_active":"2024-03-18T20:25:00.487757+0000","last_peered":"2024-03-18T20:25:00.487757+0000","last_clean":"2024-03-18T20:25:00.487757+0000","last_became_active":"2024-03-18T20:24:40.943521+0000","last_became_peered":"2024-03-18T20:24:40.943521+0000","last_unstale":"2024-03-18T20:25:00.487757+0000","last_undegraded":"2024-03-18T20:25:00.487757+0000","last_fullsized":"2024-03-18T20:25:00.487757+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:44.872498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433854+0000","last_change":"2024-03-18T20:24:41.358139+0000","last_active":"2024-03-18T20:24:45.433854+0000","last_peered":"2024-03-18T20:24:45.433854+0000","last_clean":"2024-03-18T20:24:45.433854+0000","last_became_active":"2024-03-18T20:24:41.357789+0000","last_became_peered":"2024-03-18T20:24:41.357789+0000","last_unstale":"2024-03-18T20:24:45.433854+0000","last_undegraded":"2024-03-18T20:24:45.433854+0000","last_fullsized":"2024-03-18T20:24:45.433854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:49.643210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449427+0000","last_change":"2024-03-18T20:24:45.448899+0000","last_active":"2024-03-18T20:24:45.449427+0000","last_peered":"2024-03-18T20:24:45.449427+0000","last_clean":"2024-03-18T20:24:45.449427+0000","last_became_active":"2024-03-18T20:24:45.448519+0000","last_became_peered":"2024-03-18T20:24:45.448519+0000","last_unstale":"2024-03-18T20:24:45.449427+0000","last_undegraded":"2024-03-18T20:24:45.449427+0000","last_fullsized":"2024-03-18T20:24:45.449427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:54.071495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469997+0000","last_change":"2024-03-18T20:24:40.944455+0000","last_active":"2024-03-18T20:24:58.469997+0000","last_peered":"2024-03-18T20:24:58.469997+0000","last_clean":"2024-03-18T20:24:58.469997+0000","last_became_active":"2024-03-18T20:24:40.944115+0000","last_became_peered":"2024-03-18T20:24:40.944115+0000","last_unstale":"2024-03-18T20:24:58.469997+0000","last_undegraded":"2024-03-18T20:24:58.469997+0000","last_fullsized":"2024-03-18T20:24:58.469997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:06.160375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430066+0000","last_change":"2024-03-18T20:24:41.356074+0000","last_active":"2024-03-18T20:24:45.430066+0000","last_peered":"2024-03-18T20:24:45.430066+0000","last_clean":"2024-03-18T20:24:45.430066+0000","last_became_active":"2024-03-18T20:24:41.355755+0000","last_became_peered":"2024-03-18T20:24:41.355755+0000","last_unstale":"2024-03-18T20:24:45.430066+0000","last_undegraded":"2024-03-18T20:24:45.430066+0000","last_fullsized":"2024-03-18T20:24:45.430066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:54.007791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.443111+0000","last_change":"2024-03-18T20:24:45.440797+0000","last_active":"2024-03-18T20:24:45.443111+0000","last_peered":"2024-03-18T20:24:45.443111+0000","last_clean":"2024-03-18T20:24:45.443111+0000","last_became_active":"2024-03-18T20:24:45.440410+0000","last_became_peered":"2024-03-18T20:24:45.440410+0000","last_unstale":"2024-03-18T20:24:45.443111+0000","last_undegraded":"2024-03-18T20:24:45.443111+0000","last_fullsized":"2024-03-18T20:24:45.443111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:02.586794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471938+0000","last_change":"2024-03-18T20:24:40.944239+0000","last_active":"2024-03-18T20:24:58.471938+0000","last_peered":"2024-03-18T20:24:58.471938+0000","last_clean":"2024-03-18T20:24:58.471938+0000","last_became_active":"2024-03-18T20:24:40.943845+0000","last_became_peered":"2024-03-18T20:24:40.943845+0000","last_unstale":"2024-03-18T20:24:58.471938+0000","last_undegraded":"2024-03-18T20:24:58.471938+0000","last_fullsized":"2024-03-18T20:24:58.471938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:30.966789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436700+0000","last_change":"2024-03-18T20:24:41.373565+0000","last_active":"2024-03-18T20:24:45.436700+0000","last_peered":"2024-03-18T20:24:45.436700+0000","last_clean":"2024-03-18T20:24:45.436700+0000","last_became_active":"2024-03-18T20:24:41.373201+0000","last_became_peered":"2024-03-18T20:24:41.373201+0000","last_unstale":"2024-03-18T20:24:45.436700+0000","last_undegraded":"2024-03-18T20:24:45.436700+0000","last_fullsized":"2024-03-18T20:24:45.436700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:13.118518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.447664+0000","last_change":"2024-03-18T20:24:45.447115+0000","last_active":"2024-03-18T20:24:45.447664+0000","last_peered":"2024-03-18T20:24:45.447664+0000","last_clean":"2024-03-18T20:24:45.447664+0000","last_became_active":"2024-03-18T20:24:45.446316+0000","last_became_peered":"2024-03-18T20:24:45.446316+0000","last_unstale":"2024-03-18T20:24:45.447664+0000","last_undegraded":"2024-03-18T20:24:45.447664+0000","last_fullsized":"2024-03-18T20:24:45.447664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:17.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:01.491134+0000","last_change":"2024-03-18T20:24:40.938678+0000","last_active":"2024-03-18T20:25:01.491134+0000","last_peered":"2024-03-18T20:25:01.491134+0000","last_clean":"2024-03-18T20:25:01.491134+0000","last_became_active":"2024-03-18T20:24:40.938300+0000","last_became_peered":"2024-03-18T20:24:40.938300+0000","last_unstale":"2024-03-18T20:25:01.491134+0000","last_undegraded":"2024-03-18T20:25:01.491134+0000","last_fullsized":"2024-03-18T20:25:01.491134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:21.806721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434871+0000","last_change":"2024-03-18T20:24:41.368977+0000","last_active":"2024-03-18T20:24:45.434871+0000","last_peered":"2024-03-18T20:24:45.434871+0000","last_clean":"2024-03-18T20:24:45.434871+0000","last_became_active":"2024-03-18T20:24:41.368545+0000","last_became_peered":"2024-03-18T20:24:41.368545+0000","last_unstale":"2024-03-18T20:24:45.434871+0000","last_undegraded":"2024-03-18T20:24:45.434871+0000","last_fullsized":"2024-03-18T20:24:45.434871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:23.827215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962771+0000","last_change":"2024-03-18T20:24:45.445759+0000","last_active":"2024-03-18T20:24:45.962771+0000","last_peered":"2024-03-18T20:24:45.962771+0000","last_clean":"2024-03-18T20:24:45.962771+0000","last_became_active":"2024-03-18T20:24:45.445366+0000","last_became_peered":"2024-03-18T20:24:45.445366+0000","last_unstale":"2024-03-18T20:24:45.962771+0000","last_undegraded":"2024-03-18T20:24:45.962771+0000","last_fullsized":"2024-03-18T20:24:45.962771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:59.103814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479110+0000","last_change":"2024-03-18T20:24:40.950980+0000","last_active":"2024-03-18T20:24:59.479110+0000","last_peered":"2024-03-18T20:24:59.479110+0000","last_clean":"2024-03-18T20:24:59.479110+0000","last_became_active":"2024-03-18T20:24:40.950280+0000","last_became_peered":"2024-03-18T20:24:40.950280+0000","last_unstale":"2024-03-18T20:24:59.479110+0000","last_undegraded":"2024-03-18T20:24:59.479110+0000","last_fullsized":"2024-03-18T20:24:59.479110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:16:08.679066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437457+0000","last_change":"2024-03-18T20:24:41.374302+0000","last_active":"2024-03-18T20:24:45.437457+0000","last_peered":"2024-03-18T20:24:45.437457+0000","last_clean":"2024-03-18T20:24:45.437457+0000","last_became_active":"2024-03-18T20:24:41.373974+0000","last_became_peered":"2024-03-18T20:24:41.373974+0000","last_unstale":"2024-03-18T20:24:45.437457+0000","last_undegraded":"2024-03-18T20:24:45.437457+0000","last_fullsized":"2024-03-18T20:24:45.437457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445064+0000","last_change":"2024-03-18T20:24:45.443527+0000","last_active":"2024-03-18T20:24:45.445064+0000","last_peered":"2024-03-18T20:24:45.445064+0000","last_clean":"2024-03-18T20:24:45.445064+0000","last_became_active":"2024-03-18T20:24:45.443105+0000","last_became_peered":"2024-03-18T20:24:45.443105+0000","last_unstale":"2024-03-18T20:24:45.445064+0000","last_undegraded":"2024-03-18T20:24:45.445064+0000","last_fullsized":"2024-03-18T20:24:45.445064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:04.568061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962307+0000","last_change":"2024-03-18T20:24:40.948223+0000","last_active":"2024-03-18T20:24:45.962307+0000","last_peered":"2024-03-18T20:24:45.962307+0000","last_clean":"2024-03-18T20:24:45.962307+0000","last_became_active":"2024-03-18T20:24:40.947889+0000","last_became_peered":"2024-03-18T20:24:40.947889+0000","last_unstale":"2024-03-18T20:24:45.962307+0000","last_undegraded":"2024-03-18T20:24:45.962307+0000","last_fullsized":"2024-03-18T20:24:45.962307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:33.804508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430232+0000","last_change":"2024-03-18T20:24:41.369441+0000","last_active":"2024-03-18T20:24:45.430232+0000","last_peered":"2024-03-18T20:24:45.430232+0000","last_clean":"2024-03-18T20:24:45.430232+0000","last_became_active":"2024-03-18T20:24:41.368271+0000","last_became_peered":"2024-03-18T20:24:41.368271+0000","last_unstale":"2024-03-18T20:24:45.430232+0000","last_undegraded":"2024-03-18T20:24:45.430232+0000","last_fullsized":"2024-03-18T20:24:45.430232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:20.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962427+0000","last_change":"2024-03-18T20:24:45.441117+0000","last_active":"2024-03-18T20:24:45.962427+0000","last_peered":"2024-03-18T20:24:45.962427+0000","last_clean":"2024-03-18T20:24:45.962427+0000","last_became_active":"2024-03-18T20:24:45.440774+0000","last_became_peered":"2024-03-18T20:24:45.440774+0000","last_unstale":"2024-03-18T20:24:45.962427+0000","last_undegraded":"2024-03-18T20:24:45.962427+0000","last_fullsized":"2024-03-18T20:24:45.962427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:27.034432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431547+0000","last_change":"2024-03-18T20:24:40.940987+0000","last_active":"2024-03-18T20:24:45.431547+0000","last_peered":"2024-03-18T20:24:45.431547+0000","last_clean":"2024-03-18T20:24:45.431547+0000","last_became_active":"2024-03-18T20:24:40.940652+0000","last_became_peered":"2024-03-18T20:24:40.940652+0000","last_unstale":"2024-03-18T20:24:45.431547+0000","last_undegraded":"2024-03-18T20:24:45.431547+0000","last_fullsized":"2024-03-18T20:24:45.431547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:57.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852466+0000","last_change":"2024-03-18T20:24:41.356381+0000","last_active":"2024-03-18T20:24:45.852466+0000","last_peered":"2024-03-18T20:24:45.852466+0000","last_clean":"2024-03-18T20:24:45.852466+0000","last_became_active":"2024-03-18T20:24:41.355078+0000","last_became_peered":"2024-03-18T20:24:41.355078+0000","last_unstale":"2024-03-18T20:24:45.852466+0000","last_undegraded":"2024-03-18T20:24:45.852466+0000","last_fullsized":"2024-03-18T20:24:45.852466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:03.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961701+0000","last_change":"2024-03-18T20:24:45.448305+0000","last_active":"2024-03-18T20:24:45.961701+0000","last_peered":"2024-03-18T20:24:45.961701+0000","last_clean":"2024-03-18T20:24:45.961701+0000","last_became_active":"2024-03-18T20:24:45.447706+0000","last_became_peered":"2024-03-18T20:24:45.447706+0000","last_unstale":"2024-03-18T20:24:45.961701+0000","last_undegraded":"2024-03-18T20:24:45.961701+0000","last_fullsized":"2024-03-18T20:24:45.961701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:22.305139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963464+0000","last_change":"2024-03-18T20:24:40.939482+0000","last_active":"2024-03-18T20:24:45.963464+0000","last_peered":"2024-03-18T20:24:45.963464+0000","last_clean":"2024-03-18T20:24:45.963464+0000","last_became_active":"2024-03-18T20:24:40.938983+0000","last_became_peered":"2024-03-18T20:24:40.938983+0000","last_unstale":"2024-03-18T20:24:45.963464+0000","last_undegraded":"2024-03-18T20:24:45.963464+0000","last_fullsized":"2024-03-18T20:24:45.963464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:00.111633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852446+0000","last_change":"2024-03-18T20:24:41.366749+0000","last_active":"2024-03-18T20:24:45.852446+0000","last_peered":"2024-03-18T20:24:45.852446+0000","last_clean":"2024-03-18T20:24:45.852446+0000","last_became_active":"2024-03-18T20:24:41.366335+0000","last_became_peered":"2024-03-18T20:24:41.366335+0000","last_unstale":"2024-03-18T20:24:45.852446+0000","last_undegraded":"2024-03-18T20:24:45.852446+0000","last_fullsized":"2024-03-18T20:24:45.852446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:36.927348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962802+0000","last_change":"2024-03-18T20:24:45.447585+0000","last_active":"2024-03-18T20:24:45.962802+0000","last_peered":"2024-03-18T20:24:45.962802+0000","last_clean":"2024-03-18T20:24:45.962802+0000","last_became_active":"2024-03-18T20:24:45.447165+0000","last_became_peered":"2024-03-18T20:24:45.447165+0000","last_unstale":"2024-03-18T20:24:45.962802+0000","last_undegraded":"2024-03-18T20:24:45.962802+0000","last_fullsized":"2024-03-18T20:24:45.962802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:14:34.960675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438282+0000","last_change":"2024-03-18T20:24:41.371969+0000","last_active":"2024-03-18T20:24:45.438282+0000","last_peered":"2024-03-18T20:24:45.438282+0000","last_clean":"2024-03-18T20:24:45.438282+0000","last_became_active":"2024-03-18T20:24:41.371516+0000","last_became_peered":"2024-03-18T20:24:41.371516+0000","last_unstale":"2024-03-18T20:24:45.438282+0000","last_undegraded":"2024-03-18T20:24:45.438282+0000","last_fullsized":"2024-03-18T20:24:45.438282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:46.761356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.485423+0000","last_change":"2024-03-18T20:24:40.940375+0000","last_active":"2024-03-18T20:25:00.485423+0000","last_peered":"2024-03-18T20:25:00.485423+0000","last_clean":"2024-03-18T20:25:00.485423+0000","last_became_active":"2024-03-18T20:24:40.940053+0000","last_became_peered":"2024-03-18T20:24:40.940053+0000","last_unstale":"2024-03-18T20:25:00.485423+0000","last_undegraded":"2024-03-18T20:25:00.485423+0000","last_fullsized":"2024-03-18T20:25:00.485423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:20:17.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444410+0000","last_change":"2024-03-18T20:24:45.443315+0000","last_active":"2024-03-18T20:24:45.444410+0000","last_peered":"2024-03-18T20:24:45.444410+0000","last_clean":"2024-03-18T20:24:45.444410+0000","last_became_active":"2024-03-18T20:24:45.442889+0000","last_became_peered":"2024-03-18T20:24:45.442889+0000","last_unstale":"2024-03-18T20:24:45.444410+0000","last_undegraded":"2024-03-18T20:24:45.444410+0000","last_fullsized":"2024-03-18T20:24:45.444410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:40.097397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425997+0000","last_change":"2024-03-18T20:24:41.368952+0000","last_active":"2024-03-18T20:24:45.425997+0000","last_peered":"2024-03-18T20:24:45.425997+0000","last_clean":"2024-03-18T20:24:45.425997+0000","last_became_active":"2024-03-18T20:24:41.368629+0000","last_became_peered":"2024-03-18T20:24:41.368629+0000","last_unstale":"2024-03-18T20:24:45.425997+0000","last_undegraded":"2024-03-18T20:24:45.425997+0000","last_fullsized":"2024-03-18T20:24:45.425997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:28:58.521712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.483637+0000","last_change":"2024-03-18T20:24:40.938841+0000","last_active":"2024-03-18T20:25:00.483637+0000","last_peered":"2024-03-18T20:25:00.483637+0000","last_clean":"2024-03-18T20:25:00.483637+0000","last_became_active":"2024-03-18T20:24:40.938464+0000","last_became_peered":"2024-03-18T20:24:40.938464+0000","last_unstale":"2024-03-18T20:25:00.483637+0000","last_undegraded":"2024-03-18T20:25:00.483637+0000","last_fullsized":"2024-03-18T20:25:00.483637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:04.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":325,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434656+0000","last_change":"2024-03-18T20:23:54.597665+0000","last_active":"2024-03-18T20:24:45.434656+0000","last_peered":"2024-03-18T20:24:45.434656+0000","last_clean":"2024-03-18T20:24:45.434656+0000","last_became_active":"2024-03-18T20:23:54.597135+0000","last_became_peered":"2024-03-18T20:23:54.597135+0000","last_unstale":"2024-03-18T20:24:45.434656+0000","last_undegraded":"2024-03-18T20:24:45.434656+0000","last_fullsized":"2024-03-18T20:24:45.434656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_clean_scrub_stamp":"2024-03-18T20:21:34.333460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:59.546138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449625+0000","last_change":"2024-03-18T20:24:45.448945+0000","last_active":"2024-03-18T20:24:45.449625+0000","last_peered":"2024-03-18T20:24:45.449625+0000","last_clean":"2024-03-18T20:24:45.449625+0000","last_became_active":"2024-03-18T20:24:45.448585+0000","last_became_peered":"2024-03-18T20:24:45.448585+0000","last_unstale":"2024-03-18T20:24:45.449625+0000","last_undegraded":"2024-03-18T20:24:45.449625+0000","last_fullsized":"2024-03-18T20:24:45.449625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:49.668118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429530+0000","last_change":"2024-03-18T20:24:41.364453+0000","last_active":"2024-03-18T20:24:45.429530+0000","last_peered":"2024-03-18T20:24:45.429530+0000","last_clean":"2024-03-18T20:24:45.429530+0000","last_became_active":"2024-03-18T20:24:41.364047+0000","last_became_peered":"2024-03-18T20:24:41.364047+0000","last_unstale":"2024-03-18T20:24:45.429530+0000","last_undegraded":"2024-03-18T20:24:45.429530+0000","last_fullsized":"2024-03-18T20:24:45.429530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:32.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476629+0000","last_change":"2024-03-18T20:24:40.939926+0000","last_active":"2024-03-18T20:24:59.476629+0000","last_peered":"2024-03-18T20:24:59.476629+0000","last_clean":"2024-03-18T20:24:59.476629+0000","last_became_active":"2024-03-18T20:24:40.939514+0000","last_became_peered":"2024-03-18T20:24:40.939514+0000","last_unstale":"2024-03-18T20:24:59.476629+0000","last_undegraded":"2024-03-18T20:24:59.476629+0000","last_fullsized":"2024-03-18T20:24:59.476629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:41.852567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442992+0000","last_change":"2024-03-18T20:24:45.439880+0000","last_active":"2024-03-18T20:24:45.442992+0000","last_peered":"2024-03-18T20:24:45.442992+0000","last_clean":"2024-03-18T20:24:45.442992+0000","last_became_active":"2024-03-18T20:24:45.439455+0000","last_became_peered":"2024-03-18T20:24:45.439455+0000","last_unstale":"2024-03-18T20:24:45.442992+0000","last_undegraded":"2024-03-18T20:24:45.442992+0000","last_fullsized":"2024-03-18T20:24:45.442992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:26.050401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470311+0000","last_change":"2024-03-18T20:24:40.947764+0000","last_active":"2024-03-18T20:24:58.470311+0000","last_peered":"2024-03-18T20:24:58.470311+0000","last_clean":"2024-03-18T20:24:58.470311+0000","last_became_active":"2024-03-18T20:24:40.946947+0000","last_became_peered":"2024-03-18T20:24:40.946947+0000","last_unstale":"2024-03-18T20:24:58.470311+0000","last_undegraded":"2024-03-18T20:24:58.470311+0000","last_fullsized":"2024-03-18T20:24:58.470311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:38.279134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431036+0000","last_change":"2024-03-18T20:24:41.352213+0000","last_active":"2024-03-18T20:24:45.431036+0000","last_peered":"2024-03-18T20:24:45.431036+0000","last_clean":"2024-03-18T20:24:45.431036+0000","last_became_active":"2024-03-18T20:24:41.351895+0000","last_became_peered":"2024-03-18T20:24:41.351895+0000","last_unstale":"2024-03-18T20:24:45.431036+0000","last_undegraded":"2024-03-18T20:24:45.431036+0000","last_fullsized":"2024-03-18T20:24:45.431036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:55.756782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440641+0000","last_change":"2024-03-18T20:24:45.439483+0000","last_active":"2024-03-18T20:24:45.440641+0000","last_peered":"2024-03-18T20:24:45.440641+0000","last_clean":"2024-03-18T20:24:45.440641+0000","last_became_active":"2024-03-18T20:24:45.436952+0000","last_became_peered":"2024-03-18T20:24:45.436952+0000","last_unstale":"2024-03-18T20:24:45.440641+0000","last_undegraded":"2024-03-18T20:24:45.440641+0000","last_fullsized":"2024-03-18T20:24:45.440641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:33.839948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.478344+0000","last_change":"2024-03-18T20:24:40.948184+0000","last_active":"2024-03-18T20:24:59.478344+0000","last_peered":"2024-03-18T20:24:59.478344+0000","last_clean":"2024-03-18T20:24:59.478344+0000","last_became_active":"2024-03-18T20:24:40.947953+0000","last_became_peered":"2024-03-18T20:24:40.947953+0000","last_unstale":"2024-03-18T20:24:59.478344+0000","last_undegraded":"2024-03-18T20:24:59.478344+0000","last_fullsized":"2024-03-18T20:24:59.478344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:51.701653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426390+0000","last_change":"2024-03-18T20:24:41.356563+0000","last_active":"2024-03-18T20:24:45.426390+0000","last_peered":"2024-03-18T20:24:45.426390+0000","last_clean":"2024-03-18T20:24:45.426390+0000","last_became_active":"2024-03-18T20:24:41.356229+0000","last_became_peered":"2024-03-18T20:24:41.356229+0000","last_unstale":"2024-03-18T20:24:45.426390+0000","last_undegraded":"2024-03-18T20:24:45.426390+0000","last_fullsized":"2024-03-18T20:24:45.426390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:20.296131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854824+0000","last_change":"2024-03-18T20:24:45.440937+0000","last_active":"2024-03-18T20:24:45.854824+0000","last_peered":"2024-03-18T20:24:45.854824+0000","last_clean":"2024-03-18T20:24:45.854824+0000","last_became_active":"2024-03-18T20:24:45.440624+0000","last_became_peered":"2024-03-18T20:24:45.440624+0000","last_unstale":"2024-03-18T20:24:45.854824+0000","last_undegraded":"2024-03-18T20:24:45.854824+0000","last_fullsized":"2024-03-18T20:24:45.854824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:41.588153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470417+0000","last_change":"2024-03-18T20:24:40.940622+0000","last_active":"2024-03-18T20:24:58.470417+0000","last_peered":"2024-03-18T20:24:58.470417+0000","last_clean":"2024-03-18T20:24:58.470417+0000","last_became_active":"2024-03-18T20:24:40.940285+0000","last_became_peered":"2024-03-18T20:24:40.940285+0000","last_unstale":"2024-03-18T20:24:58.470417+0000","last_undegraded":"2024-03-18T20:24:58.470417+0000","last_fullsized":"2024-03-18T20:24:58.470417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:03.933412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442035+0000","last_change":"2024-03-18T20:24:41.370212+0000","last_active":"2024-03-18T20:24:45.442035+0000","last_peered":"2024-03-18T20:24:45.442035+0000","last_clean":"2024-03-18T20:24:45.442035+0000","last_became_active":"2024-03-18T20:24:41.369904+0000","last_became_peered":"2024-03-18T20:24:41.369904+0000","last_unstale":"2024-03-18T20:24:45.442035+0000","last_undegraded":"2024-03-18T20:24:45.442035+0000","last_fullsized":"2024-03-18T20:24:45.442035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:13:12.982929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961796+0000","last_change":"2024-03-18T20:24:45.446935+0000","last_active":"2024-03-18T20:24:45.961796+0000","last_peered":"2024-03-18T20:24:45.961796+0000","last_clean":"2024-03-18T20:24:45.961796+0000","last_became_active":"2024-03-18T20:24:45.445971+0000","last_became_peered":"2024-03-18T20:24:45.445971+0000","last_unstale":"2024-03-18T20:24:45.961796+0000","last_undegraded":"2024-03-18T20:24:45.961796+0000","last_fullsized":"2024-03-18T20:24:45.961796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:51.292743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470479+0000","last_change":"2024-03-18T20:24:40.937625+0000","last_active":"2024-03-18T20:24:58.470479+0000","last_peered":"2024-03-18T20:24:58.470479+0000","last_clean":"2024-03-18T20:24:58.470479+0000","last_became_active":"2024-03-18T20:24:40.937275+0000","last_became_peered":"2024-03-18T20:24:40.937275+0000","last_unstale":"2024-03-18T20:24:58.470479+0000","last_undegraded":"2024-03-18T20:24:58.470479+0000","last_fullsized":"2024-03-18T20:24:58.470479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:55.747307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430187+0000","last_change":"2024-03-18T20:24:41.349528+0000","last_active":"2024-03-18T20:24:45.430187+0000","last_peered":"2024-03-18T20:24:45.430187+0000","last_clean":"2024-03-18T20:24:45.430187+0000","last_became_active":"2024-03-18T20:24:41.349113+0000","last_became_peered":"2024-03-18T20:24:41.349113+0000","last_unstale":"2024-03-18T20:24:45.430187+0000","last_undegraded":"2024-03-18T20:24:45.430187+0000","last_fullsized":"2024-03-18T20:24:45.430187+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:06.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444468+0000","last_change":"2024-03-18T20:24:45.441827+0000","last_active":"2024-03-18T20:24:45.444468+0000","last_peered":"2024-03-18T20:24:45.444468+0000","last_clean":"2024-03-18T20:24:45.444468+0000","last_became_active":"2024-03-18T20:24:45.441403+0000","last_became_peered":"2024-03-18T20:24:45.441403+0000","last_unstale":"2024-03-18T20:24:45.444468+0000","last_undegraded":"2024-03-18T20:24:45.444468+0000","last_fullsized":"2024-03-18T20:24:45.444468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:42.530843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470136+0000","last_change":"2024-03-18T20:24:40.943702+0000","last_active":"2024-03-18T20:24:58.470136+0000","last_peered":"2024-03-18T20:24:58.470136+0000","last_clean":"2024-03-18T20:24:58.470136+0000","last_became_active":"2024-03-18T20:24:40.943380+0000","last_became_peered":"2024-03-18T20:24:40.943380+0000","last_unstale":"2024-03-18T20:24:58.470136+0000","last_undegraded":"2024-03-18T20:24:58.470136+0000","last_fullsized":"2024-03-18T20:24:58.470136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:04.567157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431139+0000","last_change":"2024-03-18T20:24:41.352181+0000","last_active":"2024-03-18T20:24:45.431139+0000","last_peered":"2024-03-18T20:24:45.431139+0000","last_clean":"2024-03-18T20:24:45.431139+0000","last_became_active":"2024-03-18T20:24:41.351694+0000","last_became_peered":"2024-03-18T20:24:41.351694+0000","last_unstale":"2024-03-18T20:24:45.431139+0000","last_undegraded":"2024-03-18T20:24:45.431139+0000","last_fullsized":"2024-03-18T20:24:45.431139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:00.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:47.625415+0000","last_change":"2024-03-18T20:24:45.444567+0000","last_active":"2024-03-18T20:24:47.625415+0000","last_peered":"2024-03-18T20:24:47.625415+0000","last_clean":"2024-03-18T20:24:47.625415+0000","last_became_active":"2024-03-18T20:24:45.443996+0000","last_became_peered":"2024-03-18T20:24:45.443996+0000","last_unstale":"2024-03-18T20:24:47.625415+0000","last_undegraded":"2024-03-18T20:24:47.625415+0000","last_fullsized":"2024-03-18T20:24:47.625415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:16.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426427+0000","last_change":"2024-03-18T20:24:40.948872+0000","last_active":"2024-03-18T20:24:45.426427+0000","last_peered":"2024-03-18T20:24:45.426427+0000","last_clean":"2024-03-18T20:24:45.426427+0000","last_became_active":"2024-03-18T20:24:40.948540+0000","last_became_peered":"2024-03-18T20:24:40.948540+0000","last_unstale":"2024-03-18T20:24:45.426427+0000","last_undegraded":"2024-03-18T20:24:45.426427+0000","last_fullsized":"2024-03-18T20:24:45.426427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:57.954340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426231+0000","last_change":"2024-03-18T20:24:41.355552+0000","last_active":"2024-03-18T20:24:45.426231+0000","last_peered":"2024-03-18T20:24:45.426231+0000","last_clean":"2024-03-18T20:24:45.426231+0000","last_became_active":"2024-03-18T20:24:41.355245+0000","last_became_peered":"2024-03-18T20:24:41.355245+0000","last_unstale":"2024-03-18T20:24:45.426231+0000","last_undegraded":"2024-03-18T20:24:45.426231+0000","last_fullsized":"2024-03-18T20:24:45.426231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:25.833815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961749+0000","last_change":"2024-03-18T20:24:45.443078+0000","last_active":"2024-03-18T20:24:45.961749+0000","last_peered":"2024-03-18T20:24:45.961749+0000","last_clean":"2024-03-18T20:24:45.961749+0000","last_became_active":"2024-03-18T20:24:45.442675+0000","last_became_peered":"2024-03-18T20:24:45.442675+0000","last_unstale":"2024-03-18T20:24:45.961749+0000","last_undegraded":"2024-03-18T20:24:45.961749+0000","last_fullsized":"2024-03-18T20:24:45.961749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:32.262956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472041+0000","last_change":"2024-03-18T20:24:40.943239+0000","last_active":"2024-03-18T20:24:58.472041+0000","last_peered":"2024-03-18T20:24:58.472041+0000","last_clean":"2024-03-18T20:24:58.472041+0000","last_became_active":"2024-03-18T20:24:40.942862+0000","last_became_peered":"2024-03-18T20:24:40.942862+0000","last_unstale":"2024-03-18T20:24:58.472041+0000","last_undegraded":"2024-03-18T20:24:58.472041+0000","last_fullsized":"2024-03-18T20:24:58.472041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:42.560340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437303+0000","last_change":"2024-03-18T20:24:41.361986+0000","last_active":"2024-03-18T20:24:45.437303+0000","last_peered":"2024-03-18T20:24:45.437303+0000","last_clean":"2024-03-18T20:24:45.437303+0000","last_became_active":"2024-03-18T20:24:41.361608+0000","last_became_peered":"2024-03-18T20:24:41.361608+0000","last_unstale":"2024-03-18T20:24:45.437303+0000","last_undegraded":"2024-03-18T20:24:45.437303+0000","last_fullsized":"2024-03-18T20:24:45.437303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:44.540156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441527+0000","last_change":"2024-03-18T20:24:45.440250+0000","last_active":"2024-03-18T20:24:45.441527+0000","last_peered":"2024-03-18T20:24:45.441527+0000","last_clean":"2024-03-18T20:24:45.441527+0000","last_became_active":"2024-03-18T20:24:45.439782+0000","last_became_peered":"2024-03-18T20:24:45.439782+0000","last_unstale":"2024-03-18T20:24:45.441527+0000","last_undegraded":"2024-03-18T20:24:45.441527+0000","last_fullsized":"2024-03-18T20:24:45.441527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:58.415354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427038+0000","last_change":"2024-03-18T20:24:40.945951+0000","last_active":"2024-03-18T20:24:45.427038+0000","last_peered":"2024-03-18T20:24:45.427038+0000","last_clean":"2024-03-18T20:24:45.427038+0000","last_became_active":"2024-03-18T20:24:40.945147+0000","last_became_peered":"2024-03-18T20:24:40.945147+0000","last_unstale":"2024-03-18T20:24:45.427038+0000","last_undegraded":"2024-03-18T20:24:45.427038+0000","last_fullsized":"2024-03-18T20:24:45.427038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:45.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436691+0000","last_change":"2024-03-18T20:24:41.366752+0000","last_active":"2024-03-18T20:24:45.436691+0000","last_peered":"2024-03-18T20:24:45.436691+0000","last_clean":"2024-03-18T20:24:45.436691+0000","last_became_active":"2024-03-18T20:24:41.365949+0000","last_became_peered":"2024-03-18T20:24:41.365949+0000","last_unstale":"2024-03-18T20:24:45.436691+0000","last_undegraded":"2024-03-18T20:24:45.436691+0000","last_fullsized":"2024-03-18T20:24:45.436691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:42.785307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962639+0000","last_change":"2024-03-18T20:24:45.446303+0000","last_active":"2024-03-18T20:24:45.962639+0000","last_peered":"2024-03-18T20:24:45.962639+0000","last_clean":"2024-03-18T20:24:45.962639+0000","last_became_active":"2024-03-18T20:24:45.445873+0000","last_became_peered":"2024-03-18T20:24:45.445873+0000","last_unstale":"2024-03-18T20:24:45.962639+0000","last_undegraded":"2024-03-18T20:24:45.962639+0000","last_fullsized":"2024-03-18T20:24:45.962639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:41.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438335+0000","last_change":"2024-03-18T20:24:40.945490+0000","last_active":"2024-03-18T20:24:45.438335+0000","last_peered":"2024-03-18T20:24:45.438335+0000","last_clean":"2024-03-18T20:24:45.438335+0000","last_became_active":"2024-03-18T20:24:40.945155+0000","last_became_peered":"2024-03-18T20:24:40.945155+0000","last_unstale":"2024-03-18T20:24:45.438335+0000","last_undegraded":"2024-03-18T20:24:45.438335+0000","last_fullsized":"2024-03-18T20:24:45.438335+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:50.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436214+0000","last_change":"2024-03-18T20:24:41.369354+0000","last_active":"2024-03-18T20:24:45.436214+0000","last_peered":"2024-03-18T20:24:45.436214+0000","last_clean":"2024-03-18T20:24:45.436214+0000","last_became_active":"2024-03-18T20:24:41.369079+0000","last_became_peered":"2024-03-18T20:24:41.369079+0000","last_unstale":"2024-03-18T20:24:45.436214+0000","last_undegraded":"2024-03-18T20:24:45.436214+0000","last_fullsized":"2024-03-18T20:24:45.436214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:25.264037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445168+0000","last_change":"2024-03-18T20:24:45.442906+0000","last_active":"2024-03-18T20:24:45.445168+0000","last_peered":"2024-03-18T20:24:45.445168+0000","last_clean":"2024-03-18T20:24:45.445168+0000","last_became_active":"2024-03-18T20:24:45.442318+0000","last_became_peered":"2024-03-18T20:24:45.442318+0000","last_unstale":"2024-03-18T20:24:45.445168+0000","last_undegraded":"2024-03-18T20:24:45.445168+0000","last_fullsized":"2024-03-18T20:24:45.445168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:50.656133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430336+0000","last_change":"2024-03-18T20:24:40.940614+0000","last_active":"2024-03-18T20:24:45.430336+0000","last_peered":"2024-03-18T20:24:45.430336+0000","last_clean":"2024-03-18T20:24:45.430336+0000","last_became_active":"2024-03-18T20:24:40.940089+0000","last_became_peered":"2024-03-18T20:24:40.940089+0000","last_unstale":"2024-03-18T20:24:45.430336+0000","last_undegraded":"2024-03-18T20:24:45.430336+0000","last_fullsized":"2024-03-18T20:24:45.430336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:09.264556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436344+0000","last_change":"2024-03-18T20:24:41.364100+0000","last_active":"2024-03-18T20:24:45.436344+0000","last_peered":"2024-03-18T20:24:45.436344+0000","last_clean":"2024-03-18T20:24:45.436344+0000","last_became_active":"2024-03-18T20:24:41.363696+0000","last_became_peered":"2024-03-18T20:24:41.363696+0000","last_unstale":"2024-03-18T20:24:45.436344+0000","last_undegraded":"2024-03-18T20:24:45.436344+0000","last_fullsized":"2024-03-18T20:24:45.436344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:46.070917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431227+0000","last_change":"2024-03-18T20:24:41.368853+0000","last_active":"2024-03-18T20:24:45.431227+0000","last_peered":"2024-03-18T20:24:45.431227+0000","last_clean":"2024-03-18T20:24:45.431227+0000","last_became_active":"2024-03-18T20:24:41.368568+0000","last_became_peered":"2024-03-18T20:24:41.368568+0000","last_unstale":"2024-03-18T20:24:45.431227+0000","last_undegraded":"2024-03-18T20:24:45.431227+0000","last_fullsized":"2024-03-18T20:24:45.431227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:24.063361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426949+0000","last_change":"2024-03-18T20:24:41.369055+0000","last_active":"2024-03-18T20:24:45.426949+0000","last_peered":"2024-03-18T20:24:45.426949+0000","last_clean":"2024-03-18T20:24:45.426949+0000","last_became_active":"2024-03-18T20:24:41.368726+0000","last_became_peered":"2024-03-18T20:24:41.368726+0000","last_unstale":"2024-03-18T20:24:45.426949+0000","last_undegraded":"2024-03-18T20:24:45.426949+0000","last_fullsized":"2024-03-18T20:24:45.426949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:15.454756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426328+0000","last_change":"2024-03-18T20:24:41.358811+0000","last_active":"2024-03-18T20:24:45.426328+0000","last_peered":"2024-03-18T20:24:45.426328+0000","last_clean":"2024-03-18T20:24:45.426328+0000","last_became_active":"2024-03-18T20:24:41.357776+0000","last_became_peered":"2024-03-18T20:24:41.357776+0000","last_unstale":"2024-03-18T20:24:45.426328+0000","last_undegraded":"2024-03-18T20:24:45.426328+0000","last_fullsized":"2024-03-18T20:24:45.426328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.069338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442460+0000","last_change":"2024-03-18T20:24:41.370288+0000","last_active":"2024-03-18T20:24:45.442460+0000","last_peered":"2024-03-18T20:24:45.442460+0000","last_clean":"2024-03-18T20:24:45.442460+0000","last_became_active":"2024-03-18T20:24:41.370008+0000","last_became_peered":"2024-03-18T20:24:41.370008+0000","last_unstale":"2024-03-18T20:24:45.442460+0000","last_undegraded":"2024-03-18T20:24:45.442460+0000","last_fullsized":"2024-03-18T20:24:45.442460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:18.246575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430945+0000","last_change":"2024-03-18T20:24:41.370168+0000","last_active":"2024-03-18T20:24:45.430945+0000","last_peered":"2024-03-18T20:24:45.430945+0000","last_clean":"2024-03-18T20:24:45.430945+0000","last_became_active":"2024-03-18T20:24:41.369879+0000","last_became_peered":"2024-03-18T20:24:41.369879+0000","last_unstale":"2024-03-18T20:24:45.430945+0000","last_undegraded":"2024-03-18T20:24:45.430945+0000","last_fullsized":"2024-03-18T20:24:45.430945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:30:42.280869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430414+0000","last_change":"2024-03-18T20:24:41.369036+0000","last_active":"2024-03-18T20:24:45.430414+0000","last_peered":"2024-03-18T20:24:45.430414+0000","last_clean":"2024-03-18T20:24:45.430414+0000","last_became_active":"2024-03-18T20:24:41.368722+0000","last_became_peered":"2024-03-18T20:24:41.368722+0000","last_unstale":"2024-03-18T20:24:45.430414+0000","last_undegraded":"2024-03-18T20:24:45.430414+0000","last_fullsized":"2024-03-18T20:24:45.430414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:11.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427136+0000","last_change":"2024-03-18T20:24:41.355806+0000","last_active":"2024-03-18T20:24:45.427136+0000","last_peered":"2024-03-18T20:24:45.427136+0000","last_clean":"2024-03-18T20:24:45.427136+0000","last_became_active":"2024-03-18T20:24:41.354812+0000","last_became_peered":"2024-03-18T20:24:41.354812+0000","last_unstale":"2024-03-18T20:24:45.427136+0000","last_undegraded":"2024-03-18T20:24:45.427136+0000","last_fullsized":"2024-03-18T20:24:45.427136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:49.307113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426500+0000","last_change":"2024-03-18T20:24:41.371292+0000","last_active":"2024-03-18T20:24:45.426500+0000","last_peered":"2024-03-18T20:24:45.426500+0000","last_clean":"2024-03-18T20:24:45.426500+0000","last_became_active":"2024-03-18T20:24:41.370971+0000","last_became_peered":"2024-03-18T20:24:41.370971+0000","last_unstale":"2024-03-18T20:24:45.426500+0000","last_undegraded":"2024-03-18T20:24:45.426500+0000","last_fullsized":"2024-03-18T20:24:45.426500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:47.568840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438068+0000","last_change":"2024-03-18T20:24:41.368000+0000","last_active":"2024-03-18T20:24:45.438068+0000","last_peered":"2024-03-18T20:24:45.438068+0000","last_clean":"2024-03-18T20:24:45.438068+0000","last_became_active":"2024-03-18T20:24:41.367671+0000","last_became_peered":"2024-03-18T20:24:41.367671+0000","last_unstale":"2024-03-18T20:24:45.438068+0000","last_undegraded":"2024-03-18T20:24:45.438068+0000","last_fullsized":"2024-03-18T20:24:45.438068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:17.940008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425913+0000","last_change":"2024-03-18T20:24:41.354675+0000","last_active":"2024-03-18T20:24:45.425913+0000","last_peered":"2024-03-18T20:24:45.425913+0000","last_clean":"2024-03-18T20:24:45.425913+0000","last_became_active":"2024-03-18T20:24:41.354358+0000","last_became_peered":"2024-03-18T20:24:41.354358+0000","last_unstale":"2024-03-18T20:24:45.425913+0000","last_undegraded":"2024-03-18T20:24:45.425913+0000","last_fullsized":"2024-03-18T20:24:45.425913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:47.238810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433664+0000","last_change":"2024-03-18T20:24:41.360057+0000","last_active":"2024-03-18T20:24:45.433664+0000","last_peered":"2024-03-18T20:24:45.433664+0000","last_clean":"2024-03-18T20:24:45.433664+0000","last_became_active":"2024-03-18T20:24:41.359704+0000","last_became_peered":"2024-03-18T20:24:41.359704+0000","last_unstale":"2024-03-18T20:24:45.433664+0000","last_undegraded":"2024-03-18T20:24:45.433664+0000","last_fullsized":"2024-03-18T20:24:45.433664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:22.235537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430704+0000","last_change":"2024-03-18T20:24:41.357847+0000","last_active":"2024-03-18T20:24:45.430704+0000","last_peered":"2024-03-18T20:24:45.430704+0000","last_clean":"2024-03-18T20:24:45.430704+0000","last_became_active":"2024-03-18T20:24:41.357555+0000","last_became_peered":"2024-03-18T20:24:41.357555+0000","last_unstale":"2024-03-18T20:24:45.430704+0000","last_undegraded":"2024-03-18T20:24:45.430704+0000","last_fullsized":"2024-03-18T20:24:45.430704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:01.049713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431110+0000","last_change":"2024-03-18T20:24:41.366158+0000","last_active":"2024-03-18T20:24:45.431110+0000","last_peered":"2024-03-18T20:24:45.431110+0000","last_clean":"2024-03-18T20:24:45.431110+0000","last_became_active":"2024-03-18T20:24:41.365810+0000","last_became_peered":"2024-03-18T20:24:41.365810+0000","last_unstale":"2024-03-18T20:24:45.431110+0000","last_undegraded":"2024-03-18T20:24:45.431110+0000","last_fullsized":"2024-03-18T20:24:45.431110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:46:39.376215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7119,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":22,"num_read_kb":21,"num_write":120,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":21625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514508,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":457703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776144,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1053115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.94299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.75800000000000001}]},{"osd":9,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.044},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.73499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":9,"up_from":63,"seq":270582939668,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":458521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.376},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.33900000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.443}]},{"osd":3,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.58199999999999996}]},{"osd":8,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59999999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.627},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]}]},{"osd":8,"up_from":56,"seq":240518168599,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":458110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.441},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68200000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78300000000000003}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":7,"up_from":49,"seq":210453397531,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1049767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.499}]},{"osd":2,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.436}]},{"osd":5,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.628},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51900000000000002}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67500000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68899999999999995}]},{"osd":10,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561054,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":457793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.372,"15min":0.36899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.27400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.434},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.67900000000000005,"15min":0.63300000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":5.508,"5min":5.508,"15min":5.508},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.622}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60799999999999998}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.498}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":461225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.46899999999999997},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.495,"15min":0.47999999999999998},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66300000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71399999999999997,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.438}]},{"osd":3,"last update":"Mon Mar 18 20:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.42999999999999999},"min":{"1min":0.32000000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.878},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.72999999999999998}]},{"osd":8,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74399999999999999}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.755}]},{"osd":10,"last update":"Mon Mar 18 20:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.88300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":459181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.48999999999999999},"min":{"1min":0.33900000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.49399999999999999},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.5,"15min":0.48799999999999999},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53300000000000003,"15min":0.499},"min":{"1min":0.45700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.32800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.28699999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.499}]},{"osd":5,"last update":"Mon Mar 18 20:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76700000000000002,"5min":1.165,"15min":1.165},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.501,"15min":0.505},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.54700000000000004}]},{"osd":8,"last update":"Mon Mar 18 20:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73599999999999999}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.41099999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.621}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":461259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78700000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.80300000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63400000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:24:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.33300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.81299999999999994}]},{"osd":8,"last update":"Mon Mar 18 20:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.62}]},{"osd":9,"last update":"Mon Mar 18 20:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.60699999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.749},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.77400000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":457530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33100000000000002},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.255,"5min":0.221,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.745,"5min":1.256,"15min":1.256},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83399999999999996,"5min":1.228,"15min":1.228},"last":0.63},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":1.224,"15min":1.224},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.621}]},{"osd":5,"last update":"Mon Mar 18 20:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79700000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.498}]},{"osd":7,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.54500000000000004}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.67000000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55500000000000005}]},{"osd":10,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.452,"5min":7.452,"15min":7.452},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.72199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":458691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.49199999999999999,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51000000000000001,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.442}]},{"osd":1,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.32300000000000001},"min":{"1min":0.255,"5min":0.254,"15min":0.254},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.374},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.29699999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.496,"15min":0.48099999999999998},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.504,"15min":0.48299999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.67700000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.58399999999999996,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.439}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.58999999999999997}]},{"osd":9,"last update":"Mon Mar 18 20:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.42699999999999999}]},{"osd":10,"last update":"Mon Mar 18 20:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":37,"seq":158913789986,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1049499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.751,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.42199999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.255,"5min":0.245,"15min":0.245},"max":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.432,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.41499999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79000000000000004,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.879,"15min":0.879},"last":0.52000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.62}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.504},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67800000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.68600000000000005}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:03.988 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:25:03.989 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T20:25:03.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T20:25:03.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:03 smithi136 ceph-mon[48073]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:03 smithi136 ceph-mon[48073]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:03.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:03 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3053116688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:03.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:03 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3514191263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:04.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:04 smithi098 ceph-mon[43462]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:04.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:04 smithi098 ceph-mon[43462]: pgmap v234: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-18T20:25:04.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:04 smithi169 ceph-mon[48529]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:04.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:04 smithi169 ceph-mon[48529]: pgmap v234: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-18T20:25:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:04 smithi136 ceph-mon[48073]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:04.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:04 smithi136 ceph-mon[48073]: pgmap v234: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- 172.21.15.98:0/3350098308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc101880 msgr2=0x7f58cc108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 --2- 172.21.15.98:0/3350098308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc101880 0x7f58cc108130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b0d0 tx=0x7f58c400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- 172.21.15.98:0/3350098308 shutdown_connections 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 --2- 172.21.15.98:0/3350098308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc101880 0x7f58cc108130 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 --2- 172.21.15.98:0/3350098308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc0ffce0 0x7f58cc100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 --2- 172.21.15.98:0/3350098308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc100ee0 0x7f58cc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- 172.21.15.98:0/3350098308 >> 172.21.15.98:0/3350098308 conn(0x7f58cc0fb280 msgr2=0x7f58cc0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- 172.21.15.98:0/3350098308 shutdown_connections 2024-03-18T20:25:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- 172.21.15.98:0/3350098308 wait complete. 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 Processor -- start 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.184+0000 7f58d3346700 1 -- start start 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 0x7f58cc069f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc101880 0x7f58cc19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc19e800 con 0x7f58cc100ee0 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc19e970 con 0x7f58cc0ffce0 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc19eb30 con 0x7f58cc101880 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32786/0 (socket says 172.21.15.98:32786) 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 -- 172.21.15.98:0/240815401 learned_addr learned my addr 172.21.15.98:0/240815401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d18e3700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc101880 0x7f58cc19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d10e2700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 0x7f58cc069f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 -- 172.21.15.98:0/240815401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc101880 msgr2=0x7f58cc19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc101880 0x7f58cc19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 -- 172.21.15.98:0/240815401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 msgr2=0x7f58cc069f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 0x7f58cc069f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 -- 172.21.15.98:0/240815401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c4009d00 con 0x7f58cc100ee0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d10e2700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 0x7f58cc069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d08e1700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c920 tx=0x7f58bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc007b00 con 0x7f58cc100ee0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f58bc004cc0 con 0x7f58cc100ee0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc19ee10 con 0x7f58cc100ee0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc005590 con 0x7f58cc100ee0 2024-03-18T20:25:05.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.185+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc19f2d0 con 0x7f58cc100ee0 2024-03-18T20:25:05.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.186+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc107160 con 0x7f58cc100ee0 2024-03-18T20:25:05.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.188+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f58bc004750 con 0x7f58cc100ee0 2024-03-18T20:25:05.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.188+0000 7f58c27fc700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 0x7f58b80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.189+0000 7f58d10e2700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 0x7f58b80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.189+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f58bc0ccb90 con 0x7f58cc100ee0 2024-03-18T20:25:05.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.189+0000 7f58d10e2700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 0x7f58b80a36f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58c8005fd0 tx=0x7f58c8005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.192+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f58bc096050 con 0x7f58cc100ee0 2024-03-18T20:25:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.472+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f58cc04ea80 con 0x7f58cc100ee0 2024-03-18T20:25:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.476+0000 7f58c27fc700 1 -- 172.21.15.98:0/240815401 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f58bc01d070 con 0x7f58cc100ee0 2024-03-18T20:25:05.481 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-18T20:25:05.482 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAho/hlyHNEHBAAzydHLYmBHO+p1ESW5RldlA== 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 msgr2=0x7f58b80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 0x7f58b80a36f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58c8005fd0 tx=0x7f58c8005f00 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 msgr2=0x7f58cc19ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c920 tx=0x7f58bc00cce0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 shutdown_connections 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58cc101880 0x7f58cc19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f58b80a1230 0x7f58b80a36f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58cc0ffce0 0x7f58cc069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 --2- 172.21.15.98:0/240815401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58cc100ee0 0x7f58cc19ba90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 >> 172.21.15.98:0/240815401 conn(0x7f58cc0fb280 msgr2=0x7f58cc1042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:05.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 shutdown_connections 2024-03-18T20:25:05.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:05.479+0000 7f58d3346700 1 -- 172.21.15.98:0/240815401 wait complete. 2024-03-18T20:25:05.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:05 smithi098 ceph-mon[43462]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:05.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:05 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T20:25:05.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:05 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T20:25:05.866 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:25:05.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T20:25:05.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T20:25:05.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:05 smithi169 ceph-mon[48529]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:05.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:05 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T20:25:05.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:05 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T20:25:05.908 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T20:25:05.908 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T20:25:05.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T20:25:05.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:05 smithi136 ceph-mon[48073]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:05.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:05 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T20:25:05.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:05 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/240815401' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T20:25:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:06.694 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:06 smithi098 ceph-mon[43462]: pgmap v235: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-18T20:25:06.886 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:06 smithi169 ceph-mon[48529]: pgmap v235: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-18T20:25:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:06 smithi136 ceph-mon[48073]: pgmap v235: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-18T20:25:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.224+0000 7f92dc5a5700 1 -- 172.21.15.98:0/3832380395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d40ffca0 msgr2=0x7f92d4100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.224+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/3832380395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d40ffca0 0x7f92d4100120 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009990 tx=0x7f92c4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- 172.21.15.98:0/3832380395 shutdown_connections 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/3832380395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 0x7f92d410cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/3832380395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d4105d50 0x7f92d4106130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/3832380395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d40ffca0 0x7f92d4100120 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- 172.21.15.98:0/3832380395 >> 172.21.15.98:0/3832380395 conn(0x7f92d4076c80 msgr2=0x7f92d4077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- 172.21.15.98:0/3832380395 shutdown_connections 2024-03-18T20:25:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- 172.21.15.98:0/3832380395 wait complete. 2024-03-18T20:25:07.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 Processor -- start 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- start start 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d40ffca0 0x7f92d41970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 0x7f92d4197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d419bf80 con 0x7f92d4105d50 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d419c0f0 con 0x7f92d4100660 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.225+0000 7f92dc5a5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d419c2b0 con 0x7f92d40ffca0 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32804/0 (socket says 172.21.15.98:32804) 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 -- 172.21.15.98:0/4072205051 learned_addr learned my addr 172.21.15.98:0/4072205051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92da341700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d40ffca0 0x7f92d41970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92d9b40700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 0x7f92d4197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 -- 172.21.15.98:0/4072205051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d40ffca0 msgr2=0x7f92d41970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d40ffca0 0x7f92d41970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 -- 172.21.15.98:0/4072205051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 msgr2=0x7f92d4197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 0x7f92d4197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 -- 172.21.15.98:0/4072205051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40095f0 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dab42700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f92cc007ee0 tx=0x7f92cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc017070 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f92cc00f040 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d419c590 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc005710 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.226+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d41a3df0 con 0x7f92d4105d50 2024-03-18T20:25:07.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.227+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d4101390 con 0x7f92d4105d50 2024-03-18T20:25:07.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.228+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f92cc005870 con 0x7f92d4105d50 2024-03-18T20:25:07.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.229+0000 7f92cb7fe700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 0x7f92c00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.229+0000 7f92da341700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 0x7f92c00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.229+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f92cc0cca10 con 0x7f92d4105d50 2024-03-18T20:25:07.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.229+0000 7f92da341700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 0x7f92c00a36f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92d0005950 tx=0x7f92d000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.232+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f92cc095e50 con 0x7f92d4105d50 2024-03-18T20:25:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.480+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f92d404ea80 con 0x7f92d4105d50 2024-03-18T20:25:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.485+0000 7f92cb7fe700 1 -- 172.21.15.98:0/4072205051 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260276 (secure 0 0 0) 0x7f92cc099470 con 0x7f92d4105d50 2024-03-18T20:25:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 msgr2=0x7f92c00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 0x7f92c00a36f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92d0005950 tx=0x7f92d000a300 comp rx=0 tx=0).stop 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 msgr2=0x7f92d419b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f92cc007ee0 tx=0x7f92cc00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 shutdown_connections 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92d4105d50 0x7f92d419b970 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f92d40ffca0 0x7f92d41970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f92c00a1230 0x7f92c00a36f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 --2- 172.21.15.98:0/4072205051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100660 0x7f92d4197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.487+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 >> 172.21.15.98:0/4072205051 conn(0x7f92d4076c80 msgr2=0x7f92d4069240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.488+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 shutdown_connections 2024-03-18T20:25:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:07.488+0000 7f92dc5a5700 1 -- 172.21.15.98:0/4072205051 wait complete. 2024-03-18T20:25:07.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:07 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4072205051' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T20:25:07.874 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3114713517},{"type":"v1","addr":"172.21.15.98:6801","nonce":3114713517}]},"active_addr":"172.21.15.98:6801/3114713517","active_change":"2024-03-18T20:19:45.512657+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3930321953}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3650538312}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1714041692}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1193712487}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3780077316}]}]} 2024-03-18T20:25:07.880 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T20:25:07.881 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T20:25:07.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T20:25:07.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:07 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4072205051' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T20:25:07.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:07 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4072205051' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T20:25:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:08.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:08 smithi098 ceph-mon[43462]: pgmap v236: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-18T20:25:08.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:08 smithi169 ceph-mon[48529]: pgmap v236: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-18T20:25:08.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:08 smithi136 ceph-mon[48073]: pgmap v236: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/18398614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 msgr2=0x7f0e041048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/18398614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e041048d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009990 tx=0x7f0dec009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/18398614 shutdown_connections 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/18398614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e04104fb0 0x7f0e0410cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/18398614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e040fff70 0x7f0e04100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/18398614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e041048d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/18398614 >> 172.21.15.98:0/18398614 conn(0x7f0e04075d50 msgr2=0x7f0e04076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/18398614 shutdown_connections 2024-03-18T20:25:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.098+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/18398614 wait complete. 2024-03-18T20:25:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 Processor -- start 2024-03-18T20:25:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 -- start start 2024-03-18T20:25:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e040fff70 0x7f0e04101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e04104fb0 0x7f0e041023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041a55c0 con 0x7f0e04100920 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041a5730 con 0x7f0e040fff70 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.099+0000 7f0e0a3f5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041a58f0 con 0x7f0e04104fb0 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32828/0 (socket says 172.21.15.98:32828) 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 -- 172.21.15.98:0/1476305644 learned_addr learned my addr 172.21.15.98:0/1476305644 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e03fff700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e040fff70 0x7f0e04101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 -- 172.21.15.98:0/1476305644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e04104fb0 msgr2=0x7f0e041023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e08992700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e04104fb0 0x7f0e041023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e04104fb0 0x7f0e041023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 -- 172.21.15.98:0/1476305644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e040fff70 msgr2=0x7f0e04101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e040fff70 0x7f0e04101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 -- 172.21.15.98:0/1476305644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec0095f0 con 0x7f0e04100920 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e037fe700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0dec006010 tx=0x7f0dec005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec01d070 con 0x7f0e04100920 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0dec00fbc0 con 0x7f0e04100920 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e041a5b70 con 0x7f0e04100920 2024-03-18T20:25:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec022b20 con 0x7f0e04100920 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.100+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e041a5f80 con 0x7f0e04100920 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.101+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e041092e0 con 0x7f0e04100920 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.102+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f0dec00f730 con 0x7f0e04100920 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.103+0000 7f0e017fa700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 0x7f0df00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.103+0000 7f0e03fff700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 0x7f0df00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.103+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0dec0ce8f0 con 0x7f0e04100920 2024-03-18T20:25:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.103+0000 7f0e03fff700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 0x7f0df00a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0df4005950 tx=0x7f0df400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.106+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0dec097dc0 con 0x7f0e04100920 2024-03-18T20:25:09.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.298+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e041013b0 con 0x7f0e04100920 2024-03-18T20:25:09.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.300+0000 7f0e017fa700 1 -- 172.21.15.98:0/1476305644 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f0dec09b3e0 con 0x7f0e04100920 2024-03-18T20:25:09.305 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:09.305 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:09.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 msgr2=0x7f0df00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 0x7f0df00a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0df4005950 tx=0x7f0df400a300 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 msgr2=0x7f0e04101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0dec006010 tx=0x7f0dec005720 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 shutdown_connections 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0e04104fb0 0x7f0e041023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f0df00a11b0 0x7f0df00a3670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e040fff70 0x7f0e04101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 --2- 172.21.15.98:0/1476305644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e04100920 0x7f0e04101eb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 >> 172.21.15.98:0/1476305644 conn(0x7f0e04075d50 msgr2=0x7f0e04068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 shutdown_connections 2024-03-18T20:25:09.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:09.302+0000 7f0e0a3f5700 1 -- 172.21.15.98:0/1476305644 wait complete. 2024-03-18T20:25:09.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:09 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1476305644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:09.698 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T20:25:09.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T20:25:09.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:09 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1476305644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:09.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:09 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1476305644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:10.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:10 smithi098 ceph-mon[43462]: pgmap v237: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-18T20:25:10.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:10 smithi169 ceph-mon[48529]: pgmap v237: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-18T20:25:10.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.889+0000 7fa6e96e5700 1 -- 172.21.15.98:0/726768953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e40fff70 msgr2=0x7fa6e4100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.889+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/726768953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e40fff70 0x7fa6e4100350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009990 tx=0x7fa6d4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 -- 172.21.15.98:0/726768953 shutdown_connections 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/726768953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 0x7fa6e410cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/726768953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e40fff70 0x7fa6e4100350 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/726768953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e4100920 0x7fa6e41048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 -- 172.21.15.98:0/726768953 >> 172.21.15.98:0/726768953 conn(0x7fa6e4075d90 msgr2=0x7fa6e40761a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 -- 172.21.15.98:0/726768953 shutdown_connections 2024-03-18T20:25:10.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 -- 172.21.15.98:0/726768953 wait complete. 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.890+0000 7fa6e96e5700 1 Processor -- start 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 -- start start 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 0x7fa6e41971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 0x7fa6e419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419c0c0 con 0x7fa6e4100920 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419c230 con 0x7fa6e40fff70 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e96e5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e419c3f0 con 0x7fa6e4104f90 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32838/0 (socket says 172.21.15.98:32838) 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 -- 172.21.15.98:0/2346048721 learned_addr learned my addr 172.21.15.98:0/2346048721 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e2ffd700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 0x7fa6e41971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e37fe700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 0x7fa6e419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 -- 172.21.15.98:0/2346048721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 msgr2=0x7fa6e419bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 0x7fa6e419bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 -- 172.21.15.98:0/2346048721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 msgr2=0x7fa6e41971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 0x7fa6e41971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 -- 172.21.15.98:0/2346048721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40095f0 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e2ffd700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 0x7fa6e41971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6e27fc700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800b500 tx=0x7fa6d800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011830 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.891+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa6d8011e20 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.892+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e419c6d0 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.892+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8010db0 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.892+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e41a3f30 con 0x7fa6e4100920 2024-03-18T20:25:10.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.893+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e4109260 con 0x7fa6e4100920 2024-03-18T20:25:10.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.893+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa6d8011990 con 0x7fa6e4100920 2024-03-18T20:25:10.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.894+0000 7fa6cbfff700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 0x7fa6cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.894+0000 7fa6e2ffd700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 0x7fa6cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.894+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa6d80cc660 con 0x7fa6e4100920 2024-03-18T20:25:10.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.894+0000 7fa6e2ffd700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 0x7fa6cc0a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b5c0 tx=0x7fa6d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:10.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:10.898+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa6d8095bb0 con 0x7fa6e4100920 2024-03-18T20:25:10.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:10 smithi136 ceph-mon[48073]: pgmap v237: 129 pgs: 129 active+clean; 625 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-18T20:25:11.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.092+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6e4101390 con 0x7fa6e4100920 2024-03-18T20:25:11.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.094+0000 7fa6cbfff700 1 -- 172.21.15.98:0/2346048721 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7fa6d80b7e60 con 0x7fa6e4100920 2024-03-18T20:25:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:11.099 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 msgr2=0x7fa6cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 0x7fa6cc0a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b5c0 tx=0x7fa6d4005fd0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 msgr2=0x7fa6e4197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800b500 tx=0x7fa6d800b8c0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 shutdown_connections 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6e4104f90 0x7fa6e419bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa6cc0a11b0 0x7fa6cc0a3670 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6e40fff70 0x7fa6e41971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 --2- 172.21.15.98:0/2346048721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6e4100920 0x7fa6e4197700 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 >> 172.21.15.98:0/2346048721 conn(0x7fa6e4075d90 msgr2=0x7fa6e4069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 shutdown_connections 2024-03-18T20:25:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:11.096+0000 7fa6e96e5700 1 -- 172.21.15.98:0/2346048721 wait complete. 2024-03-18T20:25:11.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T20:25:11.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T20:25:11.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T20:25:11.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T20:25:11.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T20:25:11.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T20:25:11.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T20:25:11.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T20:25:11.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T20:25:11.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T20:25:11.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T20:25:11.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T20:25:11.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:11 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2346048721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:11.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:11 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2346048721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:11.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:11 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2346048721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:12.487 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:12.588 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:12.790 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:12 smithi098 ceph-mon[43462]: pgmap v238: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-18T20:25:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:12 smithi169 ceph-mon[48529]: pgmap v238: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-18T20:25:12.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:12 smithi136 ceph-mon[48073]: pgmap v238: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-18T20:25:14.149 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:14.249 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:14.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.494+0000 7fa8168b6700 1 -- 172.21.15.98:0/710953190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa810105cd0 msgr2=0x7fa810069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.494+0000 7fa8168b6700 1 --2- 172.21.15.98:0/710953190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa810105cd0 0x7fa810069410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa800009990 tx=0x7fa800009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 -- 172.21.15.98:0/710953190 shutdown_connections 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 --2- 172.21.15.98:0/710953190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8101037e0 0x7fa810103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 --2- 172.21.15.98:0/710953190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa810105cd0 0x7fa810069410 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 --2- 172.21.15.98:0/710953190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa810069950 0x7fa810069db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 -- 172.21.15.98:0/710953190 >> 172.21.15.98:0/710953190 conn(0x7fa810076cb0 msgr2=0x7fa8100770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 -- 172.21.15.98:0/710953190 shutdown_connections 2024-03-18T20:25:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.495+0000 7fa8168b6700 1 -- 172.21.15.98:0/710953190 wait complete. 2024-03-18T20:25:14.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 Processor -- start 2024-03-18T20:25:14.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 -- start start 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa810069950 0x7fa81019d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 0x7fa81019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa810105cd0 0x7fa8101a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101a1600 con 0x7fa8101037e0 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101a1770 con 0x7fa810069950 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa8168b6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101a1930 con 0x7fa810105cd0 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa80f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 0x7fa81019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa80f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 0x7fa81019dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57758/0 (socket says 172.21.15.98:57758) 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.496+0000 7fa80f7fe700 1 -- 172.21.15.98:0/1373280259 learned_addr learned my addr 172.21.15.98:0/1373280259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa814e53700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa810105cd0 0x7fa8101a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80ffff700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa810069950 0x7fa81019d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 -- 172.21.15.98:0/1373280259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa810105cd0 msgr2=0x7fa8101a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa810105cd0 0x7fa8101a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 -- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa810069950 msgr2=0x7fa81019d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa810069950 0x7fa81019d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000095f0 con 0x7fa8101037e0 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80f7fe700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 0x7fa81019dd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b530 tx=0x7fa80400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804011830 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8101a1c10 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa804011e20 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804010db0 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.497+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8101aa560 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.498+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa810117510 con 0x7fa8101037e0 2024-03-18T20:25:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.499+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa804011990 con 0x7fa8101037e0 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80d7fa700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa7f80a11b0 0x7fa7f80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80ffff700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa7f80a11b0 0x7fa7f80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa8040cca90 con 0x7fa8101037e0 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80d7fa700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] conn(0x7fa7f80abd30 0x7fa7f80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7f80ae840 con 0x7fa7f80abd30 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.500+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8040ccea0 con 0x7fa8101037e0 2024-03-18T20:25:14.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.501+0000 7fa80ffff700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa7f80a11b0 0x7fa7f80a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b5c0 tx=0x7fa800005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.501+0000 7fa814e53700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] conn(0x7fa7f80abd30 0x7fa7f80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.501+0000 7fa814e53700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] conn(0x7fa7f80abd30 0x7fa7f80ae170 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.502+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== osd.4 v2:172.21.15.136:6808/4061852500 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa7f80ae840 con 0x7fa7f80abd30 2024-03-18T20:25:14.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.518+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 --> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa810102820 con 0x7fa7f80abd30 2024-03-18T20:25:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa80d7fa700 1 -- 172.21.15.98:0/1373280259 <== osd.4 v2:172.21.15.136:6808/4061852500 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa810102820 con 0x7fa7f80abd30 2024-03-18T20:25:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] conn(0x7fa7f80abd30 msgr2=0x7fa7f80ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] conn(0x7fa7f80abd30 0x7fa7f80ae170 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa7f80a11b0 msgr2=0x7fa7f80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa7f80a11b0 0x7fa7f80a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa80000b5c0 tx=0x7fa800005c40 comp rx=0 tx=0).stop 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 msgr2=0x7fa81019dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 --2- 172.21.15.98:0/1373280259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8101037e0 0x7fa81019dd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b530 tx=0x7fa80400b840 comp rx=0 tx=0).stop 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa814e53700 1 -- 172.21.15.98:0/1373280259 reap_dead start 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 shutdown_connections 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 >> 172.21.15.98:0/1373280259 conn(0x7fa810076cb0 msgr2=0x7fa810101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.519+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 shutdown_connections 2024-03-18T20:25:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.520+0000 7fa8168b6700 1 -- 172.21.15.98:0/1373280259 wait complete. 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 -- 172.21.15.98:0/14115387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec681000f0 msgr2=0x7fec6810db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 --2- 172.21.15.98:0/14115387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec681000f0 0x7fec6810db80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fec6000b0a0 tx=0x7fec6000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 -- 172.21.15.98:0/14115387 shutdown_connections 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 --2- 172.21.15.98:0/14115387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec681000f0 0x7fec6810db80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 --2- 172.21.15.98:0/14115387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec68105e90 0x7fec68106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 --2- 172.21.15.98:0/14115387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec680ff750 0x7fec680ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 -- 172.21.15.98:0/14115387 >> 172.21.15.98:0/14115387 conn(0x7fec68074fd0 msgr2=0x7fec680753e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 -- 172.21.15.98:0/14115387 shutdown_connections 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.560+0000 7fec6f839700 1 -- 172.21.15.98:0/14115387 wait complete. 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 Processor -- start 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 -- start start 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 0x7fec68197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec681000f0 0x7fec681977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec68105e90 0x7fec6819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6819c170 con 0x7fec680ff750 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6819c2e0 con 0x7fec681000f0 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6f839700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6819c4a0 con 0x7fec68105e90 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6d5d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 0x7fec68197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6d5d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 0x7fec68197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57782/0 (socket says 172.21.15.98:57782) 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6d5d5700 1 -- 172.21.15.98:0/2586056525 learned_addr learned my addr 172.21.15.98:0/2586056525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.561+0000 7fec6ddd6700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec68105e90 0x7fec6819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 -- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec68105e90 msgr2=0x7fec6819bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec68105e90 0x7fec6819bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 -- 172.21.15.98:0/2586056525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec681000f0 msgr2=0x7fec681977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6cdd4700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec681000f0 0x7fec681977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec681000f0 0x7fec681977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60009d00 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6d5d5700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 0x7fec68197270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec6400d980 tx=0x7fec6400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400de60 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec640044b0 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64003b40 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6819c780 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.562+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec681a3fe0 con 0x7fec680ff750 2024-03-18T20:25:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.563+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec680681f0 con 0x7fec680ff750 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.564+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fec64003ca0 con 0x7fec680ff750 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec5e7fc700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fec540a11b0 0x7fec540a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fec640cd0c0 con 0x7fec680ff750 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec6cdd4700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fec540a11b0 0x7fec540a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec5e7fc700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] conn(0x7fec540abd30 0x7fec540ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec540ae840 con 0x7fec540abd30 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec640cd430 con 0x7fec680ff750 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec6cdd4700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fec540a11b0 0x7fec540a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fec681987d0 tx=0x7fec58005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.565+0000 7fec6ddd6700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] conn(0x7fec540abd30 0x7fec540ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.566+0000 7fec6ddd6700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] conn(0x7fec540abd30 0x7fec540ae170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.567+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== osd.5 v2:172.21.15.169:6808/123568177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fec540ae840 con 0x7fec540abd30 2024-03-18T20:25:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.583+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 --> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec68110e60 con 0x7fec540abd30 2024-03-18T20:25:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.584+0000 7fec5e7fc700 1 -- 172.21.15.98:0/2586056525 <== osd.5 v2:172.21.15.169:6808/123568177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fec68110e60 con 0x7fec540abd30 2024-03-18T20:25:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] conn(0x7fec540abd30 msgr2=0x7fec540ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] conn(0x7fec540abd30 0x7fec540ae170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fec540a11b0 msgr2=0x7fec540a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fec540a11b0 0x7fec540a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fec681987d0 tx=0x7fec58005cf0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 msgr2=0x7fec68197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 --2- 172.21.15.98:0/2586056525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec680ff750 0x7fec68197270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec6400d980 tx=0x7fec6400dd40 comp rx=0 tx=0).stop 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6ddd6700 1 -- 172.21.15.98:0/2586056525 reap_dead start 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 shutdown_connections 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 >> 172.21.15.98:0/2586056525 conn(0x7fec68074fd0 msgr2=0x7fec680fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 shutdown_connections 2024-03-18T20:25:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.585+0000 7fec6f839700 1 -- 172.21.15.98:0/2586056525 wait complete. 2024-03-18T20:25:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:14.836 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:14 smithi098 ceph-mon[43462]: pgmap v239: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-18T20:25:14.854 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:14.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:14 smithi136 ceph-mon[48073]: pgmap v239: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-18T20:25:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 -- 172.21.15.98:0/847784283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76680690f0 msgr2=0x7f766810dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 --2- 172.21.15.98:0/847784283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76680690f0 0x7f766810dc30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f766000b0a0 tx=0x7f766000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 -- 172.21.15.98:0/847784283 shutdown_connections 2024-03-18T20:25:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 --2- 172.21.15.98:0/847784283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76680690f0 0x7f766810dc30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 --2- 172.21.15.98:0/847784283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7668106df0 0x7f76681071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 --2- 172.21.15.98:0/847784283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 0x7f7668068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.885+0000 7f76706c4700 1 -- 172.21.15.98:0/847784283 >> 172.21.15.98:0/847784283 conn(0x7f7668075d50 msgr2=0x7f7668076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- 172.21.15.98:0/847784283 shutdown_connections 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- 172.21.15.98:0/847784283 wait complete. 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 Processor -- start 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- start start 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 0x7f76681972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76680690f0 0x7f7668197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 0x7f766819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766819c1f0 con 0x7f7668106df0 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766819c360 con 0x7f7668068750 2024-03-18T20:25:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f76706c4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766819c520 con 0x7f76680690f0 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f766ec61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 0x7f766819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.886+0000 7f766ec61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 0x7f766819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57810/0 (socket says 172.21.15.98:57810) 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 -- 172.21.15.98:0/1711307364 learned_addr learned my addr 172.21.15.98:0/1711307364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766dc5f700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76680690f0 0x7f7668197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 -- 172.21.15.98:0/1711307364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76680690f0 msgr2=0x7f7668197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76680690f0 0x7f7668197830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 -- 172.21.15.98:0/1711307364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 msgr2=0x7f76681972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766e460700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 0x7f76681972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 0x7f76681972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660009d00 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766e460700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7668068750 0x7f76681972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f766ec61700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 0x7f766819bbe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f766000b830 tx=0x7f7660009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766000e040 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7660004e60 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766819c7a0 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660005400 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.887+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76681a4060 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.888+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f766819ca80 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.889+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f766001bb70 con 0x7f7668106df0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f765f7fe700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76540a11b0 0x7f76540a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f766e460700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76540a11b0 0x7f76540a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f76600ced00 con 0x7f7668106df0 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f765f7fe700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] conn(0x7f76540abd30 0x7f76540ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76540ae840 con 0x7f76540abd30 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76600cf110 con 0x7f7668106df0 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f766dc5f700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] conn(0x7f76540abd30 0x7f76540ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f766e460700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76540a11b0 0x7f76540a3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7664007900 tx=0x7f7664008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.890+0000 7f766dc5f700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] conn(0x7f76540abd30 0x7f76540ae170 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.891+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== osd.0 v2:172.21.15.98:6802/363521185 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f76540ae840 con 0x7f76540abd30 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.907+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 --> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7668110f20 con 0x7f76540abd30 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f765f7fe700 1 -- 172.21.15.98:0/1711307364 <== osd.0 v2:172.21.15.98:6802/363521185 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7668110f20 con 0x7f76540abd30 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] conn(0x7f76540abd30 msgr2=0x7f76540ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] conn(0x7f76540abd30 0x7f76540ae170 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76540a11b0 msgr2=0x7f76540a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f76540a11b0 0x7f76540a3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7664007900 tx=0x7f7664008040 comp rx=0 tx=0).stop 2024-03-18T20:25:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 msgr2=0x7f766819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 --2- 172.21.15.98:0/1711307364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7668106df0 0x7f766819bbe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f766000b830 tx=0x7f7660009430 comp rx=0 tx=0).stop 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f766ec61700 1 -- 172.21.15.98:0/1711307364 reap_dead start 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 shutdown_connections 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 >> 172.21.15.98:0/1711307364 conn(0x7f7668075d50 msgr2=0x7f76680febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 shutdown_connections 2024-03-18T20:25:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:14.908+0000 7f76706c4700 1 -- 172.21.15.98:0/1711307364 wait complete. 2024-03-18T20:25:15.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:14 smithi169 ceph-mon[48529]: pgmap v239: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-18T20:25:15.592 INFO:teuthology.orchestra.run.smithi098.stdout:137438953513 2024-03-18T20:25:15.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T20:25:15.890 INFO:teuthology.orchestra.run.smithi098.stdout:158913789989 2024-03-18T20:25:15.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T20:25:16.090 INFO:teuthology.orchestra.run.smithi098.stdout:34359738423 2024-03-18T20:25:16.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T20:25:16.160 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:16.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:16.213 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:16.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:16 smithi098 ceph-mon[43462]: pgmap v240: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-18T20:25:16.968 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:16.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:16 smithi136 ceph-mon[48073]: pgmap v240: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-18T20:25:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:16 smithi169 ceph-mon[48529]: pgmap v240: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-18T20:25:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- 172.21.15.98:0/1576498089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f00fff70 msgr2=0x7f22f0100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1576498089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f00fff70 0x7f22f0100350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009990 tx=0x7f22e0009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- 172.21.15.98:0/1576498089 shutdown_connections 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1576498089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22f0104fb0 0x7f22f010cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1576498089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f00fff70 0x7f22f0100350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1576498089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22f0100920 0x7f22f01048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- 172.21.15.98:0/1576498089 >> 172.21.15.98:0/1576498089 conn(0x7f22f0075d50 msgr2=0x7f22f0076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- 172.21.15.98:0/1576498089 shutdown_connections 2024-03-18T20:25:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- 172.21.15.98:0/1576498089 wait complete. 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 Processor -- start 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.452+0000 7f22f63f7700 1 -- start start 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22f00fff70 0x7f22f0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22f0100920 0x7f22f0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 0x7f22f0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f01a32d0 con 0x7f22f0104fb0 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f01a3440 con 0x7f22f00fff70 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f63f7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f01a3600 con 0x7f22f0100920 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 0x7f22f0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 0x7f22f0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57836/0 (socket says 172.21.15.98:57836) 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 -- 172.21.15.98:0/1456533509 learned_addr learned my addr 172.21.15.98:0/1456533509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22effff700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22f00fff70 0x7f22f0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22ef7fe700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22f0100920 0x7f22f0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 -- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22f0100920 msgr2=0x7f22f0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22f0100920 0x7f22f0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 -- 172.21.15.98:0/1456533509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22f00fff70 msgr2=0x7f22f0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22f00fff70 0x7f22f0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00095f0 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22f4994700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 0x7f22f0198a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ef30 tx=0x7f22e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.453+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8018070 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.454+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01a3880 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.454+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f22e800f040 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.454+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800cc70 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.454+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f01a3d80 con 0x7f22f0104fb0 2024-03-18T20:25:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.455+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22f010fd40 con 0x7f22f0104fb0 2024-03-18T20:25:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f22e8007500 con 0x7f22f0104fb0 2024-03-18T20:25:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f22d80a1230 0x7f22d80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22effff700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f22d80a1230 0x7f22d80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f22e80cc7a0 con 0x7f22f0104fb0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] conn(0x7f22d80abdb0 0x7f22d80ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d80ae8c0 con 0x7f22d80abdb0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.456+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22e80ced80 con 0x7f22f0104fb0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.457+0000 7f22effff700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f22d80a1230 0x7f22d80a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006010 tx=0x7f22e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.457+0000 7f22ef7fe700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] conn(0x7f22d80abdb0 0x7f22d80ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.457+0000 7f22ef7fe700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] conn(0x7f22d80abdb0 0x7f22d80ae1f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.458+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== osd.2 v2:172.21.15.169:6800/151861743 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f22d80ae8c0 con 0x7f22d80abdb0 2024-03-18T20:25:17.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.474+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 --> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f22f01013b0 con 0x7f22d80abdb0 2024-03-18T20:25:17.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.474+0000 7f22ed7fa700 1 -- 172.21.15.98:0/1456533509 <== osd.2 v2:172.21.15.169:6800/151861743 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f22f01013b0 con 0x7f22d80abdb0 2024-03-18T20:25:17.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] conn(0x7f22d80abdb0 msgr2=0x7f22d80ae1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] conn(0x7f22d80abdb0 0x7f22d80ae1f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f22d80a1230 msgr2=0x7f22d80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f22d80a1230 0x7f22d80a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006010 tx=0x7f22e0005960 comp rx=0 tx=0).stop 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 msgr2=0x7f22f0198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 --2- 172.21.15.98:0/1456533509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22f0104fb0 0x7f22f0198a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ef30 tx=0x7f22e800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f4994700 1 -- 172.21.15.98:0/1456533509 reap_dead start 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 shutdown_connections 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 >> 172.21.15.98:0/1456533509 conn(0x7f22f0075d50 msgr2=0x7f22f00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 shutdown_connections 2024-03-18T20:25:17.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.475+0000 7f22f63f7700 1 -- 172.21.15.98:0/1456533509 wait complete. 2024-03-18T20:25:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 -- 172.21.15.98:0/3068949482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c068750 msgr2=0x7f2a3c068bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3068949482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c068750 0x7f2a3c068bb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009990 tx=0x7f2a24009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 -- 172.21.15.98:0/3068949482 shutdown_connections 2024-03-18T20:25:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3068949482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c0690f0 0x7f2a3c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3068949482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c106df0 0x7f2a3c1071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3068949482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c068750 0x7f2a3c068bb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 -- 172.21.15.98:0/3068949482 >> 172.21.15.98:0/3068949482 conn(0x7f2a3c075d50 msgr2=0x7f2a3c076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 -- 172.21.15.98:0/3068949482 shutdown_connections 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.927+0000 7f2a4117d700 1 -- 172.21.15.98:0/3068949482 wait complete. 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 Processor -- start 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 -- start start 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c068750 0x7f2a3c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0690f0 0x7f2a3c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 0x7f2a3c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19bff0 con 0x7f2a3c106df0 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19c160 con 0x7f2a3c0690f0 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a4117d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19c320 con 0x7f2a3c068750 2024-03-18T20:25:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.928+0000 7f2a3b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 0x7f2a3c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 0x7f2a3c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57856/0 (socket says 172.21.15.98:57856) 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 -- 172.21.15.98:0/3464446028 learned_addr learned my addr 172.21.15.98:0/3464446028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3a59c700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0690f0 0x7f2a3c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3ad9d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c068750 0x7f2a3c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 -- 172.21.15.98:0/3464446028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c068750 msgr2=0x7f2a3c1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c068750 0x7f2a3c1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 -- 172.21.15.98:0/3464446028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0690f0 msgr2=0x7f2a3c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0690f0 0x7f2a3c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240095f0 con 0x7f2a3c106df0 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3ad9d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a3c068750 0x7f2a3c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a3b59e700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 0x7f2a3c19b9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e940 tx=0x7f2a2c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c0097b0 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c19c600 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2a2c00ee90 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c005450 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.929+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c1a3e60 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.930+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a3c110de0 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.931+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f2a2c0055b0 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a33fff700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a280a11b0 0x7f2a280a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a3ad9d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a280a11b0 0x7f2a280a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2a2c014070 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a33fff700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] conn(0x7f2a280abd30 0x7f2a280ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a280ae840 con 0x7f2a280abd30 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a2c0cd4e0 con 0x7f2a3c106df0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a3a59c700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] conn(0x7f2a280abd30 0x7f2a280ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a3ad9d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a280a11b0 0x7f2a280a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009c50 tx=0x7f2a34009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.932+0000 7f2a3a59c700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] conn(0x7f2a280abd30 0x7f2a280ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.933+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== osd.9 v2:172.21.15.98:6826/1957300736 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2a280ae840 con 0x7f2a280abd30 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.949+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 --> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a3c1024a0 con 0x7f2a280abd30 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.949+0000 7f2a33fff700 1 -- 172.21.15.98:0/3464446028 <== osd.9 v2:172.21.15.98:6826/1957300736 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2a3c1024a0 con 0x7f2a280abd30 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] conn(0x7f2a280abd30 msgr2=0x7f2a280ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] conn(0x7f2a280abd30 0x7f2a280ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a280a11b0 msgr2=0x7f2a280a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a280a11b0 0x7f2a280a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009c50 tx=0x7f2a34009040 comp rx=0 tx=0).stop 2024-03-18T20:25:18.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 msgr2=0x7f2a3c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 --2- 172.21.15.98:0/3464446028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a3c106df0 0x7f2a3c19b9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e940 tx=0x7f2a2c00ec50 comp rx=0 tx=0).stop 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a3b59e700 1 -- 172.21.15.98:0/3464446028 reap_dead start 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 shutdown_connections 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 >> 172.21.15.98:0/3464446028 conn(0x7f2a3c075d50 msgr2=0x7f2a3c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 shutdown_connections 2024-03-18T20:25:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:17.950+0000 7f2a4117d700 1 -- 172.21.15.98:0/3464446028 wait complete. 2024-03-18T20:25:18.863 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:18 smithi098 ceph-mon[43462]: pgmap v241: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2024-03-18T20:25:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:77309411375 2024-03-18T20:25:18.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T20:25:18.982 INFO:teuthology.orchestra.run.smithi098.stdout:270582939672 2024-03-18T20:25:18.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T20:25:18.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:18 smithi136 ceph-mon[48073]: pgmap v241: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2024-03-18T20:25:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:18 smithi169 ceph-mon[48529]: pgmap v241: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2024-03-18T20:25:19.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 -- 172.21.15.98:0/1858458472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 msgr2=0x7f37a40698a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 --2- 172.21.15.98:0/1858458472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a40698a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3798009990 tx=0x7f3798009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 -- 172.21.15.98:0/1858458472 shutdown_connections 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 --2- 172.21.15.98:0/1858458472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4069de0 0x7f37a410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 --2- 172.21.15.98:0/1858458472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37a4106e30 0x7f37a4107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 --2- 172.21.15.98:0/1858458472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a40698a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 -- 172.21.15.98:0/1858458472 >> 172.21.15.98:0/1858458472 conn(0x7f37a4075d90 msgr2=0x7f37a40761a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 -- 172.21.15.98:0/1858458472 shutdown_connections 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.202+0000 7f37a8e94700 1 -- 172.21.15.98:0/1858458472 wait complete. 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.203+0000 7f37a8e94700 1 Processor -- start 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.203+0000 7f37a8e94700 1 -- start start 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a4197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37a4069de0 0x7f37a4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4106e30 0x7f37a419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a419c0b0 con 0x7f37a4069440 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a419c220 con 0x7f37a4069de0 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a8e94700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a419c3e0 con 0x7f37a4106e30 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a4197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a4197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57878/0 (socket says 172.21.15.98:57878) 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 -- 172.21.15.98:0/2025452101 learned_addr learned my addr 172.21.15.98:0/2025452101 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 -- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4106e30 msgr2=0x7f37a419baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a2d9d700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4106e30 0x7f37a419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4106e30 0x7f37a419baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 -- 172.21.15.98:0/2025452101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37a4069de0 msgr2=0x7f37a4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37a4069de0 0x7f37a4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37980095f0 con 0x7f37a4069440 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a2d9d700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37a4106e30 0x7f37a419baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:19.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37a259c700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a4197200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f379400e9a0 tx=0x7f379400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.204+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794009790 con 0x7f37a4069440 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.205+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f379400cbd0 con 0x7f37a4069440 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.205+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794018680 con 0x7f37a4069440 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.205+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a419c6c0 con 0x7f37a4069440 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.205+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a41a3f20 con 0x7f37a4069440 2024-03-18T20:25:19.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.206+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f37a4110f10 con 0x7f37a4069440 2024-03-18T20:25:19.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.208+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f379400cd40 con 0x7f37a4069440 2024-03-18T20:25:19.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.208+0000 7f37937fe700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f378c0a11b0 0x7f378c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.208+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3794014070 con 0x7f37a4069440 2024-03-18T20:25:19.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.208+0000 7f37a1d9b700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f378c0a11b0 0x7f378c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.209+0000 7f37937fe700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] conn(0x7f378c0abd30 0x7f378c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.209+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c0ae840 con 0x7f378c0abd30 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.209+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37940cd670 con 0x7f37a4069440 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.209+0000 7f37a1d9b700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f378c0a11b0 0x7f378c0a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37980098e0 tx=0x7f3798005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.209+0000 7f37a2d9d700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] conn(0x7f378c0abd30 0x7f378c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.210+0000 7f37a2d9d700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] conn(0x7f378c0abd30 0x7f378c0ae170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.210+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== osd.8 v2:172.21.15.169:6816/3298544672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f378c0ae840 con 0x7f378c0abd30 2024-03-18T20:25:19.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.227+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 --> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f37a40614b0 con 0x7f378c0abd30 2024-03-18T20:25:19.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37937fe700 1 -- 172.21.15.98:0/2025452101 <== osd.8 v2:172.21.15.169:6816/3298544672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f37a40614b0 con 0x7f378c0abd30 2024-03-18T20:25:19.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] conn(0x7f378c0abd30 msgr2=0x7f378c0ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] conn(0x7f378c0abd30 0x7f378c0ae170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f378c0a11b0 msgr2=0x7f378c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f378c0a11b0 0x7f378c0a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37980098e0 tx=0x7f3798005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 msgr2=0x7f37a4197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a8e94700 1 --2- 172.21.15.98:0/2025452101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37a4069440 0x7f37a4197200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f379400e9a0 tx=0x7f379400ed60 comp rx=0 tx=0).stop 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.228+0000 7f37a2d9d700 1 -- 172.21.15.98:0/2025452101 reap_dead start 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.229+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 shutdown_connections 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.229+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 >> 172.21.15.98:0/2025452101 conn(0x7f37a4075d90 msgr2=0x7f37a40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.229+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 shutdown_connections 2024-03-18T20:25:19.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.229+0000 7f37a8e94700 1 -- 172.21.15.98:0/2025452101 wait complete. 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 -- 172.21.15.98:0/1104071013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238105d50 msgr2=0x7fe238106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 --2- 172.21.15.98:0/1104071013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238105d50 0x7fe238106130 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe228009990 tx=0x7fe228009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 -- 172.21.15.98:0/1104071013 shutdown_connections 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 --2- 172.21.15.98:0/1104071013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe238100660 0x7fe23810cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 --2- 172.21.15.98:0/1104071013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238105d50 0x7fe238106130 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 --2- 172.21.15.98:0/1104071013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2380ffca0 0x7fe238100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 -- 172.21.15.98:0/1104071013 >> 172.21.15.98:0/1104071013 conn(0x7fe238076c80 msgr2=0x7fe238077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 -- 172.21.15.98:0/1104071013 shutdown_connections 2024-03-18T20:25:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.832+0000 7fe23f708700 1 -- 172.21.15.98:0/1104071013 wait complete. 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 Processor -- start 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 -- start start 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2380ffca0 0x7fe238197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 0x7fe2381976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe238105d50 0x7fe23819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23819c040 con 0x7fe238100660 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23819c1b0 con 0x7fe238105d50 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23f708700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23819c370 con 0x7fe2380ffca0 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 0x7fe2381976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 0x7fe2381976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57928/0 (socket says 172.21.15.98:57928) 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 -- 172.21.15.98:0/1987683395 learned_addr learned my addr 172.21.15.98:0/1987683395 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23d4a4700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2380ffca0 0x7fe238197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23dca5700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe238105d50 0x7fe23819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 -- 172.21.15.98:0/1987683395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2380ffca0 msgr2=0x7fe238197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2380ffca0 0x7fe238197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 -- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe238105d50 msgr2=0x7fe23819ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe238105d50 0x7fe23819ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.833+0000 7fe23cca3700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280095f0 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe23cca3700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 0x7fe2381976d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe23400b530 tx=0x7fe23400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234011830 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23819c650 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe234011e20 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23400f2e0 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2381a3f30 con 0x7fe238100660 2024-03-18T20:25:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.834+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe23810fe50 con 0x7fe238100660 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.836+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe234011990 con 0x7fe238100660 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.836+0000 7fe22e7fc700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2240a11b0 0x7fe2240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe2340ccc40 con 0x7fe238100660 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe22e7fc700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] conn(0x7fe2240abd30 0x7fe2240ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2240ae840 con 0x7fe2240abd30 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2340cd050 con 0x7fe238100660 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe23dca5700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] conn(0x7fe2240abd30 0x7fe2240ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe23d4a4700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2240a11b0 0x7fe2240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.837+0000 7fe23dca5700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] conn(0x7fe2240abd30 0x7fe2240ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.838+0000 7fe23d4a4700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2240a11b0 0x7fe2240a3670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe22800b5c0 tx=0x7fe228005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.838+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== osd.7 v2:172.21.15.136:6816/1282488647 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2240ae840 con 0x7fe2240abd30 2024-03-18T20:25:19.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.854+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 --> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe238104ce0 con 0x7fe2240abd30 2024-03-18T20:25:19.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.855+0000 7fe22e7fc700 1 -- 172.21.15.98:0/1987683395 <== osd.7 v2:172.21.15.136:6816/1282488647 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe238104ce0 con 0x7fe2240abd30 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] conn(0x7fe2240abd30 msgr2=0x7fe2240ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] conn(0x7fe2240abd30 0x7fe2240ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2240a11b0 msgr2=0x7fe2240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe2240a11b0 0x7fe2240a3670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe22800b5c0 tx=0x7fe228005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 msgr2=0x7fe2381976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 --2- 172.21.15.98:0/1987683395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe238100660 0x7fe2381976d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe23400b530 tx=0x7fe23400b840 comp rx=0 tx=0).stop 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23dca5700 1 -- 172.21.15.98:0/1987683395 reap_dead start 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 shutdown_connections 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 >> 172.21.15.98:0/1987683395 conn(0x7fe238076c80 msgr2=0x7fe238069240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 shutdown_connections 2024-03-18T20:25:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.856+0000 7fe23f708700 1 -- 172.21.15.98:0/1987683395 wait complete. 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 -- 172.21.15.98:0/4010429564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860105d50 msgr2=0x7f9860106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 --2- 172.21.15.98:0/4010429564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860105d50 0x7f9860106130 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9850009990 tx=0x7f9850009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 -- 172.21.15.98:0/4010429564 shutdown_connections 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 --2- 172.21.15.98:0/4010429564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9860100660 0x7f986010cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 --2- 172.21.15.98:0/4010429564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860105d50 0x7f9860106130 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 --2- 172.21.15.98:0/4010429564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98600ffca0 0x7f9860100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 -- 172.21.15.98:0/4010429564 >> 172.21.15.98:0/4010429564 conn(0x7f9860076c80 msgr2=0x7f9860077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 -- 172.21.15.98:0/4010429564 shutdown_connections 2024-03-18T20:25:19.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.946+0000 7f9865a87700 1 -- 172.21.15.98:0/4010429564 wait complete. 2024-03-18T20:25:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 Processor -- start 2024-03-18T20:25:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 -- start start 2024-03-18T20:25:19.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98600ffca0 0x7f9860199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 0x7f98601997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9860105d50 0x7f986019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986019e170 con 0x7f9860100660 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986019e2e0 con 0x7f98600ffca0 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.947+0000 7f9865a87700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986019e4a0 con 0x7f9860105d50 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 0x7f98601997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 0x7f98601997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57942/0 (socket says 172.21.15.98:57942) 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 -- 172.21.15.98:0/3917622946 learned_addr learned my addr 172.21.15.98:0/3917622946 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985f7fe700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98600ffca0 0x7f9860199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985ffff700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9860105d50 0x7f986019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 -- 172.21.15.98:0/3917622946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9860105d50 msgr2=0x7f986019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9860105d50 0x7f986019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 -- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98600ffca0 msgr2=0x7f9860199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98600ffca0 0x7f9860199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500095f0 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985effd700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 0x7f98601997b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f985400e9a0 tx=0x7f985400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854009790 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986019e780 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9854010430 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985400cea0 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.948+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98601a5fe0 con 0x7f9860100660 2024-03-18T20:25:19.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.949+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f986010fca0 con 0x7f9860100660 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.950+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f98540105a0 con 0x7f9860100660 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985cff9700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98480a1230 0x7f98480a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9854014070 con 0x7f9860100660 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985f7fe700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98480a1230 0x7f98480a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985cff9700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] conn(0x7f98480abdb0 0x7f98480ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98480ae8c0 con 0x7f98480abdb0 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.951+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98540cd420 con 0x7f9860100660 2024-03-18T20:25:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.952+0000 7f985f7fe700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98480a1230 0x7f98480a36f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9850006010 tx=0x7f9850005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.952+0000 7f985ffff700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] conn(0x7f98480abdb0 0x7f98480ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.952+0000 7f985ffff700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] conn(0x7f98480abdb0 0x7f98480ae1f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.953+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== osd.1 v2:172.21.15.136:6800/1522927033 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f98480ae8c0 con 0x7f98480abdb0 2024-03-18T20:25:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.970+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 --> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9860069970 con 0x7f98480abdb0 2024-03-18T20:25:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.970+0000 7f985cff9700 1 -- 172.21.15.98:0/3917622946 <== osd.1 v2:172.21.15.136:6800/1522927033 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9860069970 con 0x7f98480abdb0 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.970+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] conn(0x7f98480abdb0 msgr2=0x7f98480ae1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.970+0000 7f9865a87700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] conn(0x7f98480abdb0 0x7f98480ae1f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98480a1230 msgr2=0x7f98480a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98480a1230 0x7f98480a36f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9850006010 tx=0x7f9850005960 comp rx=0 tx=0).stop 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 msgr2=0x7f98601997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 --2- 172.21.15.98:0/3917622946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9860100660 0x7f98601997b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f985400e9a0 tx=0x7f985400ecb0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f985ffff700 1 -- 172.21.15.98:0/3917622946 reap_dead start 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 shutdown_connections 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 >> 172.21.15.98:0/3917622946 conn(0x7f9860076c80 msgr2=0x7f9860069020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 shutdown_connections 2024-03-18T20:25:19.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.971+0000 7f9865a87700 1 -- 172.21.15.98:0/3917622946 wait complete. 2024-03-18T20:25:20.139 INFO:teuthology.orchestra.run.smithi098.stdout:240518168603 2024-03-18T20:25:20.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 -- 172.21.15.98:0/4065307689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 msgr2=0x7f2584105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 --2- 172.21.15.98:0/4065307689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f2584105940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2580009990 tx=0x7f2580009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 -- 172.21.15.98:0/4065307689 shutdown_connections 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 --2- 172.21.15.98:0/4065307689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2584100110 0x7f258410c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 --2- 172.21.15.98:0/4065307689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f2584105940 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 --2- 172.21.15.98:0/4065307689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25840ff750 0x7f25840ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 -- 172.21.15.98:0/4065307689 >> 172.21.15.98:0/4065307689 conn(0x7f2584074fd0 msgr2=0x7f25840753e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 -- 172.21.15.98:0/4065307689 shutdown_connections 2024-03-18T20:25:20.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.421+0000 7f258bcd9700 1 -- 172.21.15.98:0/4065307689 wait complete. 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 Processor -- start 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 -- start start 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25840ff750 0x7f25841a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2584100110 0x7f25841a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f258419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258419dfd0 con 0x7f2584105560 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258419e140 con 0x7f2584100110 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258bcd9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258419e300 con 0x7f25840ff750 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258a276700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f258419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258a276700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f258419d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57970/0 (socket says 172.21.15.98:57970) 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.422+0000 7f258a276700 1 -- 172.21.15.98:0/2804126492 learned_addr learned my addr 172.21.15.98:0/2804126492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f2589a75700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25840ff750 0x7f25841a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 -- 172.21.15.98:0/2804126492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25840ff750 msgr2=0x7f25841a3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25840ff750 0x7f25841a3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 -- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2584100110 msgr2=0x7f25841a38a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2584100110 0x7f25841a38a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800095f0 con 0x7f2584105560 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258a276700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f258419d8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f257c007d70 tx=0x7f257c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c00ad40 con 0x7f2584105560 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258419e580 con 0x7f2584105560 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f257c0128b0 con 0x7f2584105560 2024-03-18T20:25:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0116b0 con 0x7f2584105560 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.423+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25841aa350 con 0x7f2584105560 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.425+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f258410fd70 con 0x7f2584105560 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.425+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f257c012a20 con 0x7f2584105560 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f257affd700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25700a11b0 0x7f25700a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f257c0ccec0 con 0x7f2584105560 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f257affd700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] conn(0x7f25700abd30 0x7f25700ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25700ae840 con 0x7f25700abd30 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f257c0cf540 con 0x7f2584105560 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.426+0000 7f2589a75700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25700a11b0 0x7f25700a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.427+0000 7f2589a75700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25700a11b0 0x7f25700a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25800059d0 tx=0x7f2580005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.427+0000 7f2589274700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] conn(0x7f25700abd30 0x7f25700ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.427+0000 7f2589274700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] conn(0x7f25700abd30 0x7f25700ae170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.428+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== osd.10 v2:172.21.15.136:6824/3492008064 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f25700ae840 con 0x7f25700abd30 2024-03-18T20:25:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.449+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 --> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25841042a0 con 0x7f25700abd30 2024-03-18T20:25:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f257affd700 1 -- 172.21.15.98:0/2804126492 <== osd.10 v2:172.21.15.136:6824/3492008064 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f25841042a0 con 0x7f25700abd30 2024-03-18T20:25:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] conn(0x7f25700abd30 msgr2=0x7f25700ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] conn(0x7f25700abd30 0x7f25700ae170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25700a11b0 msgr2=0x7f25700a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f25700a11b0 0x7f25700a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25800059d0 tx=0x7f2580005940 comp rx=0 tx=0).stop 2024-03-18T20:25:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 msgr2=0x7f258419d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 --2- 172.21.15.98:0/2804126492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2584105560 0x7f258419d8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f257c007d70 tx=0x7f257c00a340 comp rx=0 tx=0).stop 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258a276700 1 -- 172.21.15.98:0/2804126492 reap_dead start 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 shutdown_connections 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 >> 172.21.15.98:0/2804126492 conn(0x7f2584074fd0 msgr2=0x7f258410cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 shutdown_connections 2024-03-18T20:25:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.450+0000 7f258bcd9700 1 -- 172.21.15.98:0/2804126492 wait complete. 2024-03-18T20:25:20.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.708+0000 7f47e38c6700 1 -- 172.21.15.98:0/3148919306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 msgr2=0x7f47dc10dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.708+0000 7f47e38c6700 1 --2- 172.21.15.98:0/3148919306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc10dc60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f47d400b0a0 tx=0x7f47d400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 -- 172.21.15.98:0/3148919306 shutdown_connections 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 --2- 172.21.15.98:0/3148919306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc10dc60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 --2- 172.21.15.98:0/3148919306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 0x7f47dc107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 --2- 172.21.15.98:0/3148919306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 0x7f47dc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 -- 172.21.15.98:0/3148919306 >> 172.21.15.98:0/3148919306 conn(0x7f47dc077f10 msgr2=0x7f47dc078320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 -- 172.21.15.98:0/3148919306 shutdown_connections 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 -- 172.21.15.98:0/3148919306 wait complete. 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 Processor -- start 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 -- start start 2024-03-18T20:25:20.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 0x7f47dc197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.709+0000 7f47e38c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 0x7f47dc19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc19c020 con 0x7f47dc1017c0 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc19c190 con 0x7f47dc106f40 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc19c350 con 0x7f47dc100e00 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57982/0 (socket says 172.21.15.98:57982) 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 -- 172.21.15.98:0/949640021 learned_addr learned my addr 172.21.15.98:0/949640021 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e1e63700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 0x7f47dc19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e1662700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 0x7f47dc197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 -- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 msgr2=0x7f47dc197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 0x7f47dc197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 -- 172.21.15.98:0/949640021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 msgr2=0x7f47dc19ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 0x7f47dc19ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4009d00 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e1662700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47dc100e00 0x7f47dc197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e1e63700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47dc106f40 0x7f47dc19ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e0e61700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc197660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00d950 tx=0x7f47cc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc009b40 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc19c630 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47cc009ca0 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00f620 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.710+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc1a3e90 con 0x7f47dc1017c0 2024-03-18T20:25:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.711+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47dc110e30 con 0x7f47dc1017c0 2024-03-18T20:25:20.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.712+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f47cc010460 con 0x7f47dc1017c0 2024-03-18T20:25:20.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.713+0000 7f47d27fc700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47c80a11b0 0x7f47c80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.713+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47cc0cd1b0 con 0x7f47dc1017c0 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.713+0000 7f47d27fc700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] conn(0x7f47c80abd30 0x7f47c80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.713+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47c80ae840 con 0x7f47c80abd30 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.713+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47cc0cd520 con 0x7f47dc1017c0 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.714+0000 7f47e1662700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47c80a11b0 0x7f47c80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.714+0000 7f47e1e63700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] conn(0x7f47c80abd30 0x7f47c80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.714+0000 7f47e1662700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47c80a11b0 0x7f47c80a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f47d8005fd0 tx=0x7f47d8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.715+0000 7f47e1e63700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] conn(0x7f47c80abd30 0x7f47c80ae170 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.716+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== osd.11 v2:172.21.15.169:6824/2513253907 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f47c80ae840 con 0x7f47c80abd30 2024-03-18T20:25:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.732+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 --> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47dc061430 con 0x7f47c80abd30 2024-03-18T20:25:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47d27fc700 1 -- 172.21.15.98:0/949640021 <== osd.11 v2:172.21.15.169:6824/2513253907 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f47dc061430 con 0x7f47c80abd30 2024-03-18T20:25:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] conn(0x7f47c80abd30 msgr2=0x7f47c80ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] conn(0x7f47c80abd30 0x7f47c80ae170 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47c80a11b0 msgr2=0x7f47c80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f47c80a11b0 0x7f47c80a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f47d8005fd0 tx=0x7f47d8005e20 comp rx=0 tx=0).stop 2024-03-18T20:25:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 msgr2=0x7f47dc197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 --2- 172.21.15.98:0/949640021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47dc1017c0 0x7f47dc197660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00d950 tx=0x7f47cc00dd10 comp rx=0 tx=0).stop 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e1e63700 1 -- 172.21.15.98:0/949640021 reap_dead start 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 shutdown_connections 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 >> 172.21.15.98:0/949640021 conn(0x7f47dc077f10 msgr2=0x7f47dc0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 shutdown_connections 2024-03-18T20:25:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:20.733+0000 7f47e38c6700 1 -- 172.21.15.98:0/949640021 wait complete. 2024-03-18T20:25:20.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:20 smithi098 ceph-mon[43462]: pgmap v242: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T20:25:20.766 INFO:teuthology.orchestra.run.smithi098.stdout:210453397535 2024-03-18T20:25:20.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T20:25:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 -- 172.21.15.98:0/113881124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340fff70 msgr2=0x7f7734100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 --2- 172.21.15.98:0/113881124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340fff70 0x7f7734100350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7728009990 tx=0x7f7728009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 -- 172.21.15.98:0/113881124 shutdown_connections 2024-03-18T20:25:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 --2- 172.21.15.98:0/113881124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7734104fb0 0x7f773410cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 --2- 172.21.15.98:0/113881124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340fff70 0x7f7734100350 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 --2- 172.21.15.98:0/113881124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734100920 0x7f77341048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 -- 172.21.15.98:0/113881124 >> 172.21.15.98:0/113881124 conn(0x7f7734075d50 msgr2=0x7f7734076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 -- 172.21.15.98:0/113881124 shutdown_connections 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.327+0000 7f773afa0700 1 -- 172.21.15.98:0/113881124 wait complete. 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 Processor -- start 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 -- start start 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77340fff70 0x7f7734197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 0x7f77341977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734104fb0 0x7f773419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773419c180 con 0x7f7734100920 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773419c2f0 con 0x7f77340fff70 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f773afa0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773419c4b0 con 0x7f7734104fb0 2024-03-18T20:25:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f7733fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 0x7f77341977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f7733fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 0x7f77341977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57902/0 (socket says 172.21.15.98:57902) 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.328+0000 7f7733fff700 1 -- 172.21.15.98:0/1222097174 learned_addr learned my addr 172.21.15.98:0/1222097174 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7738d3c700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77340fff70 0x7f7734197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f773953d700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734104fb0 0x7f773419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 -- 172.21.15.98:0/1222097174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734104fb0 msgr2=0x7f773419bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734104fb0 0x7f773419bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 -- 172.21.15.98:0/1222097174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77340fff70 msgr2=0x7f7734197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77340fff70 0x7f7734197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280095f0 con 0x7f7734100920 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f773953d700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7734104fb0 0x7f773419bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7733fff700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 0x7f77341977c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f772000b550 tx=0x7f772000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7720011830 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7720011e20 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773419c790 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7720010dd0 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.329+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77341a3ff0 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.330+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77341013b0 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.331+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f7720011990 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7731ffb700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77240a11b0 0x7f77240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7738d3c700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77240a11b0 0x7f77240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f77200ccc90 con 0x7f7734100920 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7731ffb700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] conn(0x7f77240abd30 0x7f77240ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77240ae840 con 0x7f77240abd30 2024-03-18T20:25:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77200cd000 con 0x7f7734100920 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f773953d700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] conn(0x7f77240abd30 0x7f77240ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f7738d3c700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77240a11b0 0x7f77240a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f77280098e0 tx=0x7f772800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.332+0000 7f773953d700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] conn(0x7f77240abd30 0x7f77240ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.333+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== osd.6 v2:172.21.15.98:6818/2163470647 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f77240ae840 con 0x7f77240abd30 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 --> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f77340ff070 con 0x7f77240abd30 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f7731ffb700 1 -- 172.21.15.98:0/1222097174 <== osd.6 v2:172.21.15.98:6818/2163470647 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f77340ff070 con 0x7f77240abd30 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] conn(0x7f77240abd30 msgr2=0x7f77240ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] conn(0x7f77240abd30 0x7f77240ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77240a11b0 msgr2=0x7f77240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f77240a11b0 0x7f77240a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f77280098e0 tx=0x7f772800b540 comp rx=0 tx=0).stop 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 msgr2=0x7f77341977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 --2- 172.21.15.98:0/1222097174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7734100920 0x7f77341977c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f772000b550 tx=0x7f772000b910 comp rx=0 tx=0).stop 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773953d700 1 -- 172.21.15.98:0/1222097174 reap_dead start 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 shutdown_connections 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 >> 172.21.15.98:0/1222097174 conn(0x7f7734075d50 msgr2=0x7f7734068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 shutdown_connections 2024-03-18T20:25:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:19.349+0000 7f773afa0700 1 -- 172.21.15.98:0/1222097174 wait complete. 2024-03-18T20:25:20.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:20 smithi136 ceph-mon[48073]: pgmap v242: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T20:25:21.046 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:21.074 INFO:teuthology.orchestra.run.smithi098.stdout:55834574900 2024-03-18T20:25:21.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T20:25:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:20 smithi169 ceph-mon[48529]: pgmap v242: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T20:25:21.174 INFO:teuthology.orchestra.run.smithi098.stdout:292057776148 2024-03-18T20:25:21.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T20:25:21.598 INFO:teuthology.orchestra.run.smithi098.stdout:188978561058 2024-03-18T20:25:21.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T20:25:21.751 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:21.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.818+0000 7feed8296700 1 -- 172.21.15.98:0/3575768398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed0100660 msgr2=0x7feed010cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.818+0000 7feed8296700 1 --2- 172.21.15.98:0/3575768398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed0100660 0x7feed010cb00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7feec800b0a0 tx=0x7feec800b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 -- 172.21.15.98:0/3575768398 shutdown_connections 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 --2- 172.21.15.98:0/3575768398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed0100660 0x7feed010cb00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 --2- 172.21.15.98:0/3575768398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feed0105d50 0x7feed0106130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 --2- 172.21.15.98:0/3575768398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed00ffca0 0x7feed0100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 -- 172.21.15.98:0/3575768398 >> 172.21.15.98:0/3575768398 conn(0x7feed0076c80 msgr2=0x7feed0077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 -- 172.21.15.98:0/3575768398 shutdown_connections 2024-03-18T20:25:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 -- 172.21.15.98:0/3575768398 wait complete. 2024-03-18T20:25:21.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 Processor -- start 2024-03-18T20:25:21.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.819+0000 7feed8296700 1 -- start start 2024-03-18T20:25:21.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 0x7feed01960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed0100660 0x7feed0196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feed0105d50 0x7feed019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed019af10 con 0x7feed00ffca0 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed019b080 con 0x7feed0100660 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed019b240 con 0x7feed0105d50 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 0x7feed01960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 0x7feed01960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58000/0 (socket says 172.21.15.98:58000) 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 -- 172.21.15.98:0/378752880 learned_addr learned my addr 172.21.15.98:0/378752880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6833700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feed0105d50 0x7feed019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed5831700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed0100660 0x7feed0196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 -- 172.21.15.98:0/378752880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feed0105d50 msgr2=0x7feed019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feed0105d50 0x7feed019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 -- 172.21.15.98:0/378752880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed0100660 msgr2=0x7feed0196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed0100660 0x7feed0196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec8009d00 con 0x7feed00ffca0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed5831700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed0100660 0x7feed0196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed6032700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 0x7feed01960d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feecc00e9a0 tx=0x7feecc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc009790 con 0x7feed00ffca0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7feecc00cbd0 con 0x7feed00ffca0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed019b520 con 0x7feed00ffca0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.820+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc018680 con 0x7feed00ffca0 2024-03-18T20:25:21.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.821+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed00691b0 con 0x7feed00ffca0 2024-03-18T20:25:21.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.821+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feed010fcc0 con 0x7feed00ffca0 2024-03-18T20:25:21.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.823+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7feecc00cd40 con 0x7feed00ffca0 2024-03-18T20:25:21.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.823+0000 7feec77fe700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feebc0a33c0 0x7feebc0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.823+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feecc014070 con 0x7feed00ffca0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.823+0000 7feed5831700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feebc0a33c0 0x7feebc0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feec77fe700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] conn(0x7feebc0adf40 0x7feebc0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feebc0b0a50 con 0x7feebc0adf40 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7feecc0cd620 con 0x7feed00ffca0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feed6833700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] conn(0x7feebc0adf40 0x7feebc0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feed6833700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] conn(0x7feebc0adf40 0x7feebc0b0380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.824+0000 7feed5831700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feebc0a33c0 0x7feebc0a5880 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feec0005fd0 tx=0x7feec0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.825+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== osd.3 v2:172.21.15.98:6810/2727695950 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feebc0b0a50 con 0x7feebc0adf40 2024-03-18T20:25:21.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 --> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feed0069690 con 0x7feebc0adf40 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feec77fe700 1 -- 172.21.15.98:0/378752880 <== osd.3 v2:172.21.15.98:6810/2727695950 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feed0069690 con 0x7feebc0adf40 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] conn(0x7feebc0adf40 msgr2=0x7feebc0b0380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] conn(0x7feebc0adf40 0x7feebc0b0380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feebc0a33c0 msgr2=0x7feebc0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feebc0a33c0 0x7feebc0a5880 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feec0005fd0 tx=0x7feec0005ee0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 msgr2=0x7feed01960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed8296700 1 --2- 172.21.15.98:0/378752880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feed00ffca0 0x7feed01960d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feecc00e9a0 tx=0x7feecc00ed60 comp rx=0 tx=0).stop 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.841+0000 7feed6833700 1 -- 172.21.15.98:0/378752880 reap_dead start 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.842+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 shutdown_connections 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.842+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 >> 172.21.15.98:0/378752880 conn(0x7feed0076c80 msgr2=0x7feed00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:21.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.842+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 shutdown_connections 2024-03-18T20:25:21.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:21.842+0000 7feed8296700 1 -- 172.21.15.98:0/378752880 wait complete. 2024-03-18T20:25:22.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:22 smithi098 ceph-mon[43462]: pgmap v243: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:25:22.565 INFO:teuthology.orchestra.run.smithi098.stdout:326417514513 2024-03-18T20:25:22.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T20:25:22.717 INFO:teuthology.orchestra.run.smithi098.stdout:107374182445 2024-03-18T20:25:22.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T20:25:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:22 smithi136 ceph-mon[48073]: pgmap v243: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:25:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:22 smithi169 ceph-mon[48529]: pgmap v243: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:25:23.178 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 -- 172.21.15.98:0/128060233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8101220 msgr2=0x7fdcf810cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 --2- 172.21.15.98:0/128060233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8101220 0x7fdcf810cb80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000b0a0 tx=0x7fdcf000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 -- 172.21.15.98:0/128060233 shutdown_connections 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 --2- 172.21.15.98:0/128060233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8101220 0x7fdcf810cb80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 --2- 172.21.15.98:0/128060233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8104ce0 0x7fdcf81050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 --2- 172.21.15.98:0/128060233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 0x7fdcf8100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 -- 172.21.15.98:0/128060233 >> 172.21.15.98:0/128060233 conn(0x7fdcf8074fd0 msgr2=0x7fdcf80753e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 -- 172.21.15.98:0/128060233 shutdown_connections 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 -- 172.21.15.98:0/128060233 wait complete. 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.214+0000 7fdcff229700 1 Processor -- start 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 -- start start 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 0x7fdcf81019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 0x7fdcf8101f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf81a5560 con 0x7fdcf8104ce0 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf81a56d0 con 0x7fdcf8100860 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcff229700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf81a5890 con 0x7fdcf8101220 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58018/0 (socket says 172.21.15.98:58018) 2024-03-18T20:25:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 -- 172.21.15.98:0/1703957074 learned_addr learned my addr 172.21.15.98:0/1703957074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfcfc5700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 0x7fdcf81019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 -- 172.21.15.98:0/1703957074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 msgr2=0x7fdcf8101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcf7fff700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 0x7fdcf8101f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 0x7fdcf8101f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 -- 172.21.15.98:0/1703957074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 msgr2=0x7fdcf81019e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 0x7fdcf81019e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 -- 172.21.15.98:0/1703957074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf0009d00 con 0x7fdcf8104ce0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcf7fff700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 0x7fdcf8101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcfd7c6700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000b830 tx=0x7fdcf0009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf000e040 con 0x7fdcf8104ce0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdcf0004e60 con 0x7fdcf8104ce0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.215+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0005400 con 0x7fdcf8104ce0 2024-03-18T20:25:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.216+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf81a5b10 con 0x7fdcf8104ce0 2024-03-18T20:25:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.216+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf81a5f30 con 0x7fdcf8104ce0 2024-03-18T20:25:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.217+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf81092e0 con 0x7fdcf8104ce0 2024-03-18T20:25:23.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.218+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fdcf001bb70 con 0x7fdcf8104ce0 2024-03-18T20:25:23.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.218+0000 7fdcf5ffb700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 0x7fdce80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.218+0000 7fdcfcfc5700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 0x7fdce80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.218+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdcf00ceb70 con 0x7fdcf8104ce0 2024-03-18T20:25:23.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.219+0000 7fdcfcfc5700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 0x7fdce80a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdcec005ea0 tx=0x7fdcec005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.225+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdcf0098040 con 0x7fdcf8104ce0 2024-03-18T20:25:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.463+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdcf804ea80 con 0x7fdcf8104ce0 2024-03-18T20:25:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.463+0000 7fdcf5ffb700 1 -- 172.21.15.98:0/1703957074 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdcf009b660 con 0x7fdcf8104ce0 2024-03-18T20:25:23.469 INFO:teuthology.orchestra.run.smithi098.stdout:158913789991 2024-03-18T20:25:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 msgr2=0x7fdce80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 0x7fdce80a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdcec005ea0 tx=0x7fdcec005e30 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 msgr2=0x7fdcf8102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000b830 tx=0x7fdcf0009430 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 shutdown_connections 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf8104ce0 0x7fdcf8102460 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdce80a11b0 0x7fdce80a3670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcf8100860 0x7fdcf81019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 --2- 172.21.15.98:0/1703957074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdcf8101220 0x7fdcf8101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 >> 172.21.15.98:0/1703957074 conn(0x7fdcf8074fd0 msgr2=0x7fdcf80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 shutdown_connections 2024-03-18T20:25:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.466+0000 7fdcff229700 1 -- 172.21.15.98:0/1703957074 wait complete. 2024-03-18T20:25:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.591+0000 7fafcce97700 1 -- 172.21.15.98:0/3344255096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 msgr2=0x7fafc80ffd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.591+0000 7fafcce97700 1 --2- 172.21.15.98:0/3344255096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc80ffd90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009990 tx=0x7fafb8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 -- 172.21.15.98:0/3344255096 shutdown_connections 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 --2- 172.21.15.98:0/3344255096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc810ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 --2- 172.21.15.98:0/3344255096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc80ffd90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 --2- 172.21.15.98:0/3344255096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc8100360 0x7fafc8108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 -- 172.21.15.98:0/3344255096 >> 172.21.15.98:0/3344255096 conn(0x7fafc8074fd0 msgr2=0x7fafc80753e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 -- 172.21.15.98:0/3344255096 shutdown_connections 2024-03-18T20:25:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.592+0000 7fafcce97700 1 -- 172.21.15.98:0/3344255096 wait complete. 2024-03-18T20:25:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.593+0000 7fafcce97700 1 Processor -- start 2024-03-18T20:25:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.593+0000 7fafcce97700 1 -- start start 2024-03-18T20:25:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc8100360 0x7fafc8103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc81035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80713d0 con 0x7fafc80ff9b0 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8071540 con 0x7fafc8100360 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafcce97700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8071700 con 0x7fafc8109580 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36900/0 (socket says 172.21.15.98:36900) 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc659c700 1 -- 172.21.15.98:0/794174661 learned_addr learned my addr 172.21.15.98:0/794174661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc81035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 -- 172.21.15.98:0/794174661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc8100360 msgr2=0x7fafc8103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc8100360 0x7fafc8103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 -- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 msgr2=0x7fafc8102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc6d9d700 1 -- 172.21.15.98:0/794174661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80095f0 con 0x7fafc8109580 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.594+0000 7fafc659c700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafc6d9d700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc81035c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafc000efc0 tx=0x7fafc000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0018070 con 0x7fafc8109580 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc8071980 con 0x7fafc8109580 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fafc000f040 con 0x7fafc8109580 2024-03-18T20:25:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc000cc50 con 0x7fafc8109580 2024-03-18T20:25:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.595+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc8071e80 con 0x7fafc8109580 2024-03-18T20:25:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.596+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc80682d0 con 0x7fafc8109580 2024-03-18T20:25:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.597+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fafc0003a00 con 0x7fafc8109580 2024-03-18T20:25:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.598+0000 7fafb77fe700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 0x7fafb00a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.598+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fafc00cc910 con 0x7fafc8109580 2024-03-18T20:25:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.598+0000 7fafc659c700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 0x7fafb00a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.599+0000 7fafc659c700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 0x7fafb00a36e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fafb800b2e0 tx=0x7fafb8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.602+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fafc0095de0 con 0x7fafc8109580 2024-03-18T20:25:23.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:23 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1703957074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T20:25:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.806+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fafc8104020 con 0x7fafc8109580 2024-03-18T20:25:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.806+0000 7fafb77fe700 1 -- 172.21.15.98:0/794174661 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fafc0099400 con 0x7fafc8109580 2024-03-18T20:25:23.811 INFO:teuthology.orchestra.run.smithi098.stdout:137438953514 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 msgr2=0x7fafb00a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 0x7fafb00a36e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fafb800b2e0 tx=0x7fafb8005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 msgr2=0x7fafc81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc81035c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafc000efc0 tx=0x7fafc000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 shutdown_connections 2024-03-18T20:25:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc8109580 0x7fafc81035c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc80ff9b0 0x7fafc8102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fafb00a1220 0x7fafb00a36e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 --2- 172.21.15.98:0/794174661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc8100360 0x7fafc8103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.808+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 >> 172.21.15.98:0/794174661 conn(0x7fafc8074fd0 msgr2=0x7fafc8107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.809+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 shutdown_connections 2024-03-18T20:25:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:23.809+0000 7fafcce97700 1 -- 172.21.15.98:0/794174661 wait complete. 2024-03-18T20:25:23.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:23 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1703957074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T20:25:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:23.912 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:23.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:23 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1703957074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T20:25:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:24.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789989 got 158913789991 for osd.5 2024-03-18T20:25:24.608 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:24.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:24 smithi098 ceph-mon[43462]: pgmap v244: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:24.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:24 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/794174661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T20:25:24.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953513 got 137438953514 for osd.4 2024-03-18T20:25:24.624 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:24.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:24 smithi169 ceph-mon[48529]: pgmap v244: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:24.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:24 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/794174661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T20:25:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:24 smithi136 ceph-mon[48073]: pgmap v244: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:24 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/794174661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T20:25:25.032 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:25.032 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 -- 172.21.15.98:0/2176962996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708100e60 msgr2=0x7fd7081012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 --2- 172.21.15.98:0/2176962996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708100e60 0x7fd7081012e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd704009990 tx=0x7fd704009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 -- 172.21.15.98:0/2176962996 shutdown_connections 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 --2- 172.21.15.98:0/2176962996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708101820 0x7fd70810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 --2- 172.21.15.98:0/2176962996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708106e40 0x7fd708107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 --2- 172.21.15.98:0/2176962996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708100e60 0x7fd7081012e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 -- 172.21.15.98:0/2176962996 >> 172.21.15.98:0/2176962996 conn(0x7fd708077f10 msgr2=0x7fd708078320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 -- 172.21.15.98:0/2176962996 shutdown_connections 2024-03-18T20:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 -- 172.21.15.98:0/2176962996 wait complete. 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.422+0000 7fd70fea7700 1 Processor -- start 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 -- start start 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 0x7fd7080730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 0x7fd7080735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 0x7fd708073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7081a53d0 con 0x7fd708106e40 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7081a5540 con 0x7fd708101820 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70fea7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7081a5700 con 0x7fd708100e60 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70dc43700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 0x7fd7080730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70d442700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 0x7fd7080735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70e444700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 0x7fd708073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70dc43700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 0x7fd7080730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.98:36856/0 (socket says 172.21.15.98:36856) 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70dc43700 1 -- 172.21.15.98:0/3672213307 learned_addr learned my addr 172.21.15.98:0/3672213307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.423+0000 7fd70e444700 1 -- 172.21.15.98:0/3672213307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 msgr2=0x7fd7080730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70e444700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 0x7fd7080730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70e444700 1 -- 172.21.15.98:0/3672213307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 msgr2=0x7fd7080735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70e444700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 0x7fd7080735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70e444700 1 -- 172.21.15.98:0/3672213307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7040095f0 con 0x7fd708106e40 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70d442700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 0x7fd7080735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70e444700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 0x7fd708073b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd700007f10 tx=0x7fd70000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700009ea0 con 0x7fd708106e40 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7081a59e0 con 0x7fd708106e40 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd70000f040 con 0x7fd708106e40 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700004280 con 0x7fd708106e40 2024-03-18T20:25:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.424+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7081a5eb0 con 0x7fd708106e40 2024-03-18T20:25:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.425+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7081092d0 con 0x7fd708106e40 2024-03-18T20:25:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.426+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd70000db20 con 0x7fd708106e40 2024-03-18T20:25:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.427+0000 7fd6faffd700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 0x7fd6f40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.427+0000 7fd70dc43700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 0x7fd6f40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.427+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd7000ccf90 con 0x7fd708106e40 2024-03-18T20:25:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.428+0000 7fd70dc43700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 0x7fd6f40a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009ca0 tx=0x7fd6fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.431+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd7000963d0 con 0x7fd708106e40 2024-03-18T20:25:25.517 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 -- 172.21.15.98:0/2083912390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3624069010 msgr2=0x7f3624069490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 --2- 172.21.15.98:0/2083912390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3624069010 0x7f3624069490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f36180099c0 tx=0x7f3618009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 -- 172.21.15.98:0/2083912390 shutdown_connections 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 --2- 172.21.15.98:0/2083912390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36240699d0 0x7f362410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 --2- 172.21.15.98:0/2083912390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f36241071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 --2- 172.21.15.98:0/2083912390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3624069010 0x7f3624069490 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.625+0000 7f3629c3a700 1 -- 172.21.15.98:0/2083912390 >> 172.21.15.98:0/2083912390 conn(0x7f3624076c80 msgr2=0x7f3624077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.626+0000 7f3629c3a700 1 -- 172.21.15.98:0/2083912390 shutdown_connections 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.626+0000 7f3629c3a700 1 -- 172.21.15.98:0/2083912390 wait complete. 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.626+0000 7f3629c3a700 1 Processor -- start 2024-03-18T20:25:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.626+0000 7f3629c3a700 1 -- start start 2024-03-18T20:25:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3624069010 0x7f3624197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f362419b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362419bee0 con 0x7f36240699d0 2024-03-18T20:25:25.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362419c050 con 0x7f3624106df0 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362419c210 con 0x7f3624069010 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36960/0 (socket says 172.21.15.98:36960) 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 -- 172.21.15.98:0/3447475278 learned_addr learned my addr 172.21.15.98:0/3447475278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f36237fe700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3624069010 0x7f3624197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3623fff700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f362419b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 -- 172.21.15.98:0/3447475278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3624069010 msgr2=0x7f3624197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3624069010 0x7f3624197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 -- 172.21.15.98:0/3447475278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 msgr2=0x7f362419b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f362419b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 -- 172.21.15.98:0/3447475278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180095f0 con 0x7f36240699d0 2024-03-18T20:25:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3623fff700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f362419b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3622ffd700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3618009f50 tx=0x7f3618005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361801d070 con 0x7f36240699d0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362419c490 con 0x7f36240699d0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.627+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f361800fc00 con 0x7f36240699d0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.628+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618022ca0 con 0x7f36240699d0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.628+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36241a3d50 con 0x7f36240699d0 2024-03-18T20:25:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.629+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36241092f0 con 0x7f36240699d0 2024-03-18T20:25:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.630+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f3618022470 con 0x7f36240699d0 2024-03-18T20:25:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.630+0000 7f3620ff9700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 0x7f360c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.630+0000 7f36237fe700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 0x7f360c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.630+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f36180ce660 con 0x7f36240699d0 2024-03-18T20:25:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.631+0000 7f36237fe700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 0x7f360c0a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3614005950 tx=0x7f361400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.634+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3618097b30 con 0x7f36240699d0 2024-03-18T20:25:25.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.654+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fd708074590 con 0x7fd708106e40 2024-03-18T20:25:25.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.654+0000 7fd6faffd700 1 -- 172.21.15.98:0/3672213307 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd700019020 con 0x7fd708106e40 2024-03-18T20:25:25.659 INFO:teuthology.orchestra.run.smithi098.stdout:270582939674 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 msgr2=0x7fd6f40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 0x7fd6f40a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009ca0 tx=0x7fd6fc009260 comp rx=0 tx=0).stop 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 msgr2=0x7fd708073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 0x7fd708073b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd700007f10 tx=0x7fd70000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 shutdown_connections 2024-03-18T20:25:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd708106e40 0x7fd708073b30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd708100e60 0x7fd7080730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd6f40a11b0 0x7fd6f40a3670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 --2- 172.21.15.98:0/3672213307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd708101820 0x7fd7080735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 >> 172.21.15.98:0/3672213307 conn(0x7fd708077f10 msgr2=0x7fd7080ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 shutdown_connections 2024-03-18T20:25:25.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.657+0000 7fd70fea7700 1 -- 172.21.15.98:0/3672213307 wait complete. 2024-03-18T20:25:25.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.851+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f36241982f0 con 0x7f36240699d0 2024-03-18T20:25:25.857 INFO:teuthology.orchestra.run.smithi098.stdout:34359738425 2024-03-18T20:25:25.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.852+0000 7f3620ff9700 1 -- 172.21.15.98:0/3447475278 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f361809b150 con 0x7f36240699d0 2024-03-18T20:25:25.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 msgr2=0x7f360c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 0x7f360c0a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3614005950 tx=0x7f361400b410 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 msgr2=0x7f3624197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3618009f50 tx=0x7f3618005740 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 shutdown_connections 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f360c0a11b0 0x7f360c0a3670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3624106df0 0x7f362419b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3624069010 0x7f3624197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 --2- 172.21.15.98:0/3447475278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36240699d0 0x7f3624197570 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 >> 172.21.15.98:0/3447475278 conn(0x7f3624076c80 msgr2=0x7f36240fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 shutdown_connections 2024-03-18T20:25:25.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:25.854+0000 7f3629c3a700 1 -- 172.21.15.98:0/3447475278 wait complete. 2024-03-18T20:25:26.018 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:26.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:26 smithi098 ceph-mon[43462]: pgmap v245: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:26.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3672213307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T20:25:26.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:26 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3447475278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T20:25:26.771 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738423 got 34359738425 for osd.0 2024-03-18T20:25:26.772 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:26 smithi136 ceph-mon[48073]: pgmap v245: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3672213307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T20:25:26.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:26 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3447475278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T20:25:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:26 smithi169 ceph-mon[48529]: pgmap v245: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:27.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3672213307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T20:25:27.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:26 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3447475278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T20:25:27.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939672 got 270582939674 for osd.9 2024-03-18T20:25:27.306 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- 172.21.15.98:0/1546878095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac104fb0 msgr2=0x7f01ac10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 --2- 172.21.15.98:0/1546878095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac104fb0 0x7f01ac10cb80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f01a400b0a0 tx=0x7f01a400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- 172.21.15.98:0/1546878095 shutdown_connections 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 --2- 172.21.15.98:0/1546878095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac104fb0 0x7f01ac10cb80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 --2- 172.21.15.98:0/1546878095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac0fff70 0x7f01ac100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 --2- 172.21.15.98:0/1546878095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac100920 0x7f01ac1048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- 172.21.15.98:0/1546878095 >> 172.21.15.98:0/1546878095 conn(0x7f01ac075d50 msgr2=0x7f01ac076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- 172.21.15.98:0/1546878095 shutdown_connections 2024-03-18T20:25:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- 172.21.15.98:0/1546878095 wait complete. 2024-03-18T20:25:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 Processor -- start 2024-03-18T20:25:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.421+0000 7f01b1e75700 1 -- start start 2024-03-18T20:25:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac0fff70 0x7f01ac1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac104fb0 0x7f01ac19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac19c0d0 con 0x7f01ac100920 2024-03-18T20:25:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac19c240 con 0x7f01ac104fb0 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01b1e75700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac19c400 con 0x7f01ac0fff70 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36982/0 (socket says 172.21.15.98:36982) 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 -- 172.21.15.98:0/3941238708 learned_addr learned my addr 172.21.15.98:0/3941238708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01ab7fe700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac0fff70 0x7f01ac1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01abfff700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac104fb0 0x7f01ac19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 -- 172.21.15.98:0/3941238708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac0fff70 msgr2=0x7f01ac1971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac0fff70 0x7f01ac1971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 -- 172.21.15.98:0/3941238708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac104fb0 msgr2=0x7f01ac19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac104fb0 0x7f01ac19bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.422+0000 7f01a2ffd700 1 -- 172.21.15.98:0/3941238708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4009d00 con 0x7f01ac100920 2024-03-18T20:25:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01a2ffd700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f019400c9e0 tx=0x7f019400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194007aa0 con 0x7f01ac100920 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f019400ce80 con 0x7f01ac100920 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac19c6e0 con 0x7f01ac100920 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194018400 con 0x7f01ac100920 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.423+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac1a3f40 con 0x7f01ac100920 2024-03-18T20:25:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.424+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac1013b0 con 0x7f01ac100920 2024-03-18T20:25:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.425+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f019400f450 con 0x7f01ac100920 2024-03-18T20:25:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.426+0000 7f01a97fa700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 0x7f01980a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.426+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f01940ccde0 con 0x7f01ac100920 2024-03-18T20:25:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.426+0000 7f01ab7fe700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 0x7f01980a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.426+0000 7f01ab7fe700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 0x7f01980a36f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f019c009a10 tx=0x7f019c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.430+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0194096330 con 0x7f01ac100920 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.671+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f01ac04ea80 con 0x7f01ac100920 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.672+0000 7f01a97fa700 1 -- 172.21.15.98:0/3941238708 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0194099950 con 0x7f01ac100920 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi098.stdout:77309411377 2024-03-18T20:25:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 msgr2=0x7f01980a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 0x7f01980a36f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f019c009a10 tx=0x7f019c009190 comp rx=0 tx=0).stop 2024-03-18T20:25:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 msgr2=0x7f01ac197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f019400c9e0 tx=0x7f019400ccf0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 shutdown_connections 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f01980a1230 0x7f01980a36f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01ac104fb0 0x7f01ac19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01ac0fff70 0x7f01ac1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 --2- 172.21.15.98:0/3941238708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ac100920 0x7f01ac197710 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 >> 172.21.15.98:0/3941238708 conn(0x7f01ac075d50 msgr2=0x7f01ac0ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 shutdown_connections 2024-03-18T20:25:28.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.675+0000 7f01b1e75700 1 -- 172.21.15.98:0/3941238708 wait complete. 2024-03-18T20:25:28.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:28 smithi098 ceph-mon[43462]: pgmap v246: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:28.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/1141927744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069050 msgr2=0x7fdd680694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/1141927744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069050 0x7fdd680694d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0099c0 tx=0x7fdd5c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/1141927744 shutdown_connections 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/1141927744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069a10 0x7fdd6810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/1141927744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 0x7fdd681071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/1141927744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069050 0x7fdd680694d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/1141927744 >> 172.21.15.98:0/1141927744 conn(0x7fdd68076c80 msgr2=0x7fdd68077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/1141927744 shutdown_connections 2024-03-18T20:25:28.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.728+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/1141927744 wait complete. 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 Processor -- start 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 -- start start 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069050 0x7fdd68197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 0x7fdd6819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6819bf90 con 0x7fdd68069a10 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6819c100 con 0x7fdd68106df0 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd6e0e9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6819c2c0 con 0x7fdd68069050 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd66ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd66ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36990/0 (socket says 172.21.15.98:36990) 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd66ffd700 1 -- 172.21.15.98:0/2518916937 learned_addr learned my addr 172.21.15.98:0/2518916937 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd677fe700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069050 0x7fdd68197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.729+0000 7fdd67fff700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 0x7fdd6819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 -- 172.21.15.98:0/2518916937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069050 msgr2=0x7fdd68197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069050 0x7fdd68197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 -- 172.21.15.98:0/2518916937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 msgr2=0x7fdd6819b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 0x7fdd6819b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 -- 172.21.15.98:0/2518916937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0095f0 con 0x7fdd68069a10 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd66ffd700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009f50 tx=0x7fdd5c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c01d070 con 0x7fdd68069a10 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd5c00fb80 con 0x7fdd68069a10 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6819c540 con 0x7fdd68069a10 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c022ba0 con 0x7fdd68069a10 2024-03-18T20:25:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.730+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd681a3e00 con 0x7fdd68069a10 2024-03-18T20:25:28.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.732+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd681092f0 con 0x7fdd68069a10 2024-03-18T20:25:28.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.732+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fdd5c00f6f0 con 0x7fdd68069a10 2024-03-18T20:25:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.733+0000 7fdd64ff9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 0x7fdd500a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.733+0000 7fdd677fe700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 0x7fdd500a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.733+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd5c0cdc90 con 0x7fdd68069a10 2024-03-18T20:25:28.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.734+0000 7fdd677fe700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 0x7fdd500a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009a10 tx=0x7fdd58009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.738+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd5c05d9f0 con 0x7fdd68069a10 2024-03-18T20:25:28.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.962+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fdd6804ea80 con 0x7fdd68069a10 2024-03-18T20:25:28.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.962+0000 7fdd64ff9700 1 -- 172.21.15.98:0/2518916937 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdd5c0d3020 con 0x7fdd68069a10 2024-03-18T20:25:28.967 INFO:teuthology.orchestra.run.smithi098.stdout:240518168605 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 msgr2=0x7fdd500a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 0x7fdd500a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009a10 tx=0x7fdd58009190 comp rx=0 tx=0).stop 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 msgr2=0x7fdd681975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009f50 tx=0x7fdd5c005740 comp rx=0 tx=0).stop 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 shutdown_connections 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fdd500a1230 0x7fdd500a36f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd68106df0 0x7fdd6819b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd68069050 0x7fdd68197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 --2- 172.21.15.98:0/2518916937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd68069a10 0x7fdd681975d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.964+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 >> 172.21.15.98:0/2518916937 conn(0x7fdd68076c80 msgr2=0x7fdd680fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.965+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 shutdown_connections 2024-03-18T20:25:28.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:28.965+0000 7fdd6e0e9700 1 -- 172.21.15.98:0/2518916937 wait complete. 2024-03-18T20:25:28.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:28 smithi136 ceph-mon[48073]: pgmap v246: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 -- 172.21.15.98:0/1788902807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 msgr2=0x7fba44107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 --2- 172.21.15.98:0/1788902807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba44107200 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fba34009990 tx=0x7fba34009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 -- 172.21.15.98:0/1788902807 shutdown_connections 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 --2- 172.21.15.98:0/1788902807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 0x7fba4410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 --2- 172.21.15.98:0/1788902807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba44107200 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 --2- 172.21.15.98:0/1788902807 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 0x7fba44069870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 -- 172.21.15.98:0/1788902807 >> 172.21.15.98:0/1788902807 conn(0x7fba44076cb0 msgr2=0x7fba440770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:29.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 -- 172.21.15.98:0/1788902807 shutdown_connections 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.087+0000 7fba4a299700 1 -- 172.21.15.98:0/1788902807 wait complete. 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 Processor -- start 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 -- start start 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 0x7fba44197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 0x7fba44197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4419bff0 con 0x7fba44106e20 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4419c160 con 0x7fba44069db0 2024-03-18T20:25:29.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba4a299700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4419c320 con 0x7fba44069410 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba48836700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba48836700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37022/0 (socket says 172.21.15.98:37022) 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba43fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 0x7fba44197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.088+0000 7fba48836700 1 -- 172.21.15.98:0/2749551805 learned_addr learned my addr 172.21.15.98:0/2749551805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba437fe700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 0x7fba44197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 -- 172.21.15.98:0/2749551805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 msgr2=0x7fba44197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 0x7fba44197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 -- 172.21.15.98:0/2749551805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 msgr2=0x7fba44197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 0x7fba44197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 -- 172.21.15.98:0/2749551805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340095f0 con 0x7fba44106e20 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba48836700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fba3c007ee0 tx=0x7fba3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c017070 con 0x7fba44106e20 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4419c600 con 0x7fba44106e20 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fba3c00f040 con 0x7fba44106e20 2024-03-18T20:25:29.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c005710 con 0x7fba44106e20 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.089+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba441a3e60 con 0x7fba44106e20 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.091+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44109340 con 0x7fba44106e20 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.092+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fba3c005870 con 0x7fba44106e20 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.093+0000 7fba417fa700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 0x7fba300a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.093+0000 7fba43fff700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 0x7fba300a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.093+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fba3c0cc150 con 0x7fba44106e20 2024-03-18T20:25:29.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.093+0000 7fba43fff700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 0x7fba300a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba34006010 tx=0x7fba34005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.096+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fba3c095590 con 0x7fba44106e20 2024-03-18T20:25:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:28 smithi169 ceph-mon[48529]: pgmap v246: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:25:29.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.302+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fba441024d0 con 0x7fba44106e20 2024-03-18T20:25:29.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.302+0000 7fba417fa700 1 -- 172.21.15.98:0/2749551805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fba3c093e60 con 0x7fba44106e20 2024-03-18T20:25:29.307 INFO:teuthology.orchestra.run.smithi098.stdout:188978561060 2024-03-18T20:25:29.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 msgr2=0x7fba300a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 0x7fba300a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba34006010 tx=0x7fba34005960 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 msgr2=0x7fba4419b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fba3c007ee0 tx=0x7fba3c00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 shutdown_connections 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba44106e20 0x7fba4419b9e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fba44069410 0x7fba44197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fba300a1230 0x7fba300a36f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 --2- 172.21.15.98:0/2749551805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44069db0 0x7fba44197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 >> 172.21.15.98:0/2749551805 conn(0x7fba44076cb0 msgr2=0x7fba440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 shutdown_connections 2024-03-18T20:25:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.304+0000 7fba4a299700 1 -- 172.21.15.98:0/2749551805 wait complete. 2024-03-18T20:25:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.011+0000 7fd3487cc700 1 -- 172.21.15.98:0/1635908022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340106df0 msgr2=0x7fd3401071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.011+0000 7fd3487cc700 1 --2- 172.21.15.98:0/1635908022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340106df0 0x7fd3401071d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009990 tx=0x7fd33c009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 -- 172.21.15.98:0/1635908022 shutdown_connections 2024-03-18T20:25:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 --2- 172.21.15.98:0/1635908022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3400690f0 0x7fd34010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 --2- 172.21.15.98:0/1635908022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd340106df0 0x7fd3401071d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 --2- 172.21.15.98:0/1635908022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 0x7fd340068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 -- 172.21.15.98:0/1635908022 >> 172.21.15.98:0/1635908022 conn(0x7fd340075d50 msgr2=0x7fd340076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 -- 172.21.15.98:0/1635908022 shutdown_connections 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 -- 172.21.15.98:0/1635908022 wait complete. 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 Processor -- start 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.012+0000 7fd3487cc700 1 -- start start 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 0x7fd340197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd340106df0 0x7fd34019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34019c000 con 0x7fd3400690f0 2024-03-18T20:25:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34019c170 con 0x7fd340106df0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd3487cc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34019c330 con 0x7fd340068750 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37000/0 (socket says 172.21.15.98:37000) 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 -- 172.21.15.98:0/3891187730 learned_addr learned my addr 172.21.15.98:0/3891187730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd346d69700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd340106df0 0x7fd34019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd346568700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 0x7fd340197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 -- 172.21.15.98:0/3891187730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 msgr2=0x7fd340197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 0x7fd340197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 -- 172.21.15.98:0/3891187730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd340106df0 msgr2=0x7fd34019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd340106df0 0x7fd34019b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 -- 172.21.15.98:0/3891187730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0095f0 con 0x7fd3400690f0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.013+0000 7fd345d67700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd33000e9a0 tx=0x7fd33000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.014+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330009790 con 0x7fd3400690f0 2024-03-18T20:25:29.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.014+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34019c610 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.014+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd33000cbd0 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.014+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330018680 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.014+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3401a3e70 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.015+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd340109310 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.016+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd33000cd40 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.017+0000 7fd3377fe700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 0x7fd32c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.017+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd330014070 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.017+0000 7fd346568700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 0x7fd32c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.017+0000 7fd346568700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 0x7fd32c0a3670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd33c005f50 tx=0x7fd33c005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.021+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd3300529f0 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.233+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd340198410 con 0x7fd3400690f0 2024-03-18T20:25:29.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.233+0000 7fd3377fe700 1 -- 172.21.15.98:0/3891187730 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd330099e30 con 0x7fd3400690f0 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stdout:210453397537 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.235+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 msgr2=0x7fd32c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.235+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 0x7fd32c0a3670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd33c005f50 tx=0x7fd33c005e80 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.235+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 msgr2=0x7fd340197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.235+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd33000e9a0 tx=0x7fd33000ed60 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 shutdown_connections 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd32c0a11b0 0x7fd32c0a3670 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd340106df0 0x7fd34019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd340068750 0x7fd340197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 --2- 172.21.15.98:0/3891187730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3400690f0 0x7fd340197690 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 >> 172.21.15.98:0/3891187730 conn(0x7fd340075d50 msgr2=0x7fd3400fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 shutdown_connections 2024-03-18T20:25:29.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.236+0000 7fd3487cc700 1 -- 172.21.15.98:0/3891187730 wait complete. 2024-03-18T20:25:29.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:29 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3941238708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T20:25:29.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:29 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2518916937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T20:25:29.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:29 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3891187730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T20:25:29.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:29 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2749551805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T20:25:29.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411375 got 77309411377 for osd.2 2024-03-18T20:25:29.942 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:29 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3941238708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T20:25:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:29 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2518916937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T20:25:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:29 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3891187730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T20:25:29.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:29 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2749551805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T20:25:30.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:29 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3941238708' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T20:25:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:29 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2518916937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T20:25:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:29 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3891187730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T20:25:30.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:29 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2749551805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T20:25:30.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561058 got 188978561060 for osd.6 2024-03-18T20:25:30.198 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:30.289 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397535 got 210453397537 for osd.7 2024-03-18T20:25:30.289 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 -- 172.21.15.98:0/1877676197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 msgr2=0x7fa290068bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 --2- 172.21.15.98:0/1877676197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa290068bb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa284009990 tx=0x7fa284009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 -- 172.21.15.98:0/1877676197 shutdown_connections 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 --2- 172.21.15.98:0/1877676197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2900690f0 0x7fa29010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 --2- 172.21.15.98:0/1877676197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa290106df0 0x7fa2901071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 --2- 172.21.15.98:0/1877676197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa290068bb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 -- 172.21.15.98:0/1877676197 >> 172.21.15.98:0/1877676197 conn(0x7fa290075d50 msgr2=0x7fa290076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 -- 172.21.15.98:0/1877676197 shutdown_connections 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 -- 172.21.15.98:0/1877676197 wait complete. 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.769+0000 7fa294ab5700 1 Processor -- start 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 -- start start 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2900690f0 0x7fa2901976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa290106df0 0x7fa29019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29019c060 con 0x7fa290068750 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29019c1d0 con 0x7fa2900690f0 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa294ab5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29019c390 con 0x7fa290106df0 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37044/0 (socket says 172.21.15.98:37044) 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 -- 172.21.15.98:0/501770671 learned_addr learned my addr 172.21.15.98:0/501770671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28dd9b700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2900690f0 0x7fa2901976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28ed9d700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa290106df0 0x7fa29019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 -- 172.21.15.98:0/501770671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa290106df0 msgr2=0x7fa29019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa290106df0 0x7fa29019ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 -- 172.21.15.98:0/501770671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2900690f0 msgr2=0x7fa2901976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2900690f0 0x7fa2901976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 -- 172.21.15.98:0/501770671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2840095f0 con 0x7fa290068750 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.770+0000 7fa28e59c700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa28000d950 tx=0x7fa28000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.771+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280009940 con 0x7fa290068750 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.771+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa280010460 con 0x7fa290068750 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.771+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29019c670 con 0x7fa290068750 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.771+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28000f620 con 0x7fa290068750 2024-03-18T20:25:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.771+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2901a3ed0 con 0x7fa290068750 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.772+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa290109310 con 0x7fa290068750 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.773+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa2800105d0 con 0x7fa290068750 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.773+0000 7fa27f7fe700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 0x7fa2780a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.774+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa2800cca90 con 0x7fa290068750 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stdout:55834574902 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.774+0000 7fa28dd9b700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 0x7fa2780a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.774+0000 7fa28dd9b700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 0x7fa2780a3670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2901986c0 tx=0x7fa284005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.778+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa280095ed0 con 0x7fa290068750 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.982+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa290198470 con 0x7fa290068750 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.982+0000 7fa27f7fe700 1 -- 172.21.15.98:0/501770671 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa280016090 con 0x7fa290068750 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 msgr2=0x7fa2780a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 0x7fa2780a3670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2901986c0 tx=0x7fa284005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 msgr2=0x7fa2901971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa28000d950 tx=0x7fa28000dc60 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 shutdown_connections 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa290106df0 0x7fa29019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa290068750 0x7fa2901971b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa2780a11b0 0x7fa2780a3670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 --2- 172.21.15.98:0/501770671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2900690f0 0x7fa2901976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 >> 172.21.15.98:0/501770671 conn(0x7fa290075d50 msgr2=0x7fa2900fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 shutdown_connections 2024-03-18T20:25:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:29.984+0000 7fa294ab5700 1 -- 172.21.15.98:0/501770671 wait complete. 2024-03-18T20:25:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/3460407277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff8101c60 msgr2=0x7f2ff8069480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/3460407277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff8101c60 0x7f2ff8069480 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009990 tx=0x7f2fe8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/3460407277 shutdown_connections 2024-03-18T20:25:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/3460407277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 0x7f2ff810fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/3460407277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff8101c60 0x7f2ff8069480 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/3460407277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff80699c0 0x7f2ff8069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/3460407277 >> 172.21.15.98:0/3460407277 conn(0x7f2ff8076b40 msgr2=0x7f2ff8076f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/3460407277 shutdown_connections 2024-03-18T20:25:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.584+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/3460407277 wait complete. 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.585+0000 7f2ffe0d0700 1 Processor -- start 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.585+0000 7f2ffe0d0700 1 -- start start 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff8101c60 0x7f2ff8199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 0x7f2ff819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff819e200 con 0x7f2ff80699c0 2024-03-18T20:25:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff819e370 con 0x7f2ff8101c60 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ffe0d0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff819e530 con 0x7f2ff810d600 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37074/0 (socket says 172.21.15.98:37074) 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 -- 172.21.15.98:0/2425674615 learned_addr learned my addr 172.21.15.98:0/2425674615 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff6ffd700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff8101c60 0x7f2ff8199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff7fff700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 0x7f2ff819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 -- 172.21.15.98:0/2425674615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 msgr2=0x7f2ff819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 0x7f2ff819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 -- 172.21.15.98:0/2425674615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff8101c60 msgr2=0x7f2ff8199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff8101c60 0x7f2ff8199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 -- 172.21.15.98:0/2425674615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe80095f0 con 0x7f2ff80699c0 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.586+0000 7f2ff77fe700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006010 tx=0x7f2fe8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.587+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe801d070 con 0x7f2ff80699c0 2024-03-18T20:25:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.587+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff819e7b0 con 0x7f2ff80699c0 2024-03-18T20:25:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.587+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2fe800fb80 con 0x7f2ff80699c0 2024-03-18T20:25:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.587+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8022b90 con 0x7f2ff80699c0 2024-03-18T20:25:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.587+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff81a6070 con 0x7f2ff80699c0 2024-03-18T20:25:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.588+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff80ffe30 con 0x7f2ff80699c0 2024-03-18T20:25:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.590+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f2fe800f6f0 con 0x7f2ff80699c0 2024-03-18T20:25:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.591+0000 7f2ff4ff9700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 0x7f2fe00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.591+0000 7f2ff6ffd700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 0x7f2fe00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.591+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2fe80ce070 con 0x7f2ff80699c0 2024-03-18T20:25:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.591+0000 7f2ff6ffd700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 0x7f2fe00a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2ff819a860 tx=0x7f2fec007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.595+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2fe8097540 con 0x7f2ff80699c0 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 -- 172.21.15.98:0/2433905432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740fff70 msgr2=0x7f8474100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 --2- 172.21.15.98:0/2433905432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740fff70 0x7f8474100350 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8464009990 tx=0x7f8464009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 -- 172.21.15.98:0/2433905432 shutdown_connections 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 --2- 172.21.15.98:0/2433905432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847410cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 --2- 172.21.15.98:0/2433905432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740fff70 0x7f8474100350 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 --2- 172.21.15.98:0/2433905432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8474100920 0x7f84741048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 -- 172.21.15.98:0/2433905432 >> 172.21.15.98:0/2433905432 conn(0x7f8474075d50 msgr2=0x7f8474076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 -- 172.21.15.98:0/2433905432 shutdown_connections 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.650+0000 7f8479b8b700 1 -- 172.21.15.98:0/2433905432 wait complete. 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 Processor -- start 2024-03-18T20:25:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 -- start start 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 0x7f8474197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847419c080 con 0x7f8474100920 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847419c1f0 con 0x7f8474104fb0 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8479b8b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847419c3b0 con 0x7f84740fff70 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.651+0000 7f8472ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37102/0 (socket says 172.21.15.98:37102) 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 -- 172.21.15.98:0/3875180861 learned_addr learned my addr 172.21.15.98:0/3875180861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f84737fe700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 0x7f8474197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8473fff700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 -- 172.21.15.98:0/3875180861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 msgr2=0x7f8474197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 0x7f8474197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 -- 172.21.15.98:0/3875180861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 msgr2=0x7f847419ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847419ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 -- 172.21.15.98:0/3875180861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84640095f0 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f84737fe700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 0x7f8474197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8473fff700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847419ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8472ffd700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f846800d950 tx=0x7f846800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846800bb60 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847419c690 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8468007620 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.652+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84741a3ef0 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.653+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468004220 con 0x7f8474100920 2024-03-18T20:25:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.653+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84741013b0 con 0x7f8474100920 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.655+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f8468004380 con 0x7f8474100920 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.656+0000 7f8470ff9700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 0x7f845c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.656+0000 7f84737fe700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 0x7f845c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.656+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f84680ccdd0 con 0x7f8474100920 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.657+0000 7f84737fe700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 0x7f845c0a3670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f84640059d0 tx=0x7f8464005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.660+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8468096210 con 0x7f8474100920 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 -- 172.21.15.98:0/2863748107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc106ef0 msgr2=0x7fb3fc1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 --2- 172.21.15.98:0/2863748107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc1072d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009990 tx=0x7fb3ec009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 -- 172.21.15.98:0/2863748107 shutdown_connections 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 --2- 172.21.15.98:0/2863748107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc10dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 --2- 172.21.15.98:0/2863748107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc1072d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 --2- 172.21.15.98:0/2863748107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc0ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.410+0000 7fb400dec700 1 -- 172.21.15.98:0/2863748107 >> 172.21.15.98:0/2863748107 conn(0x7fb3fc075e80 msgr2=0x7fb3fc0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.411+0000 7fb400dec700 1 -- 172.21.15.98:0/2863748107 shutdown_connections 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.411+0000 7fb400dec700 1 -- 172.21.15.98:0/2863748107 wait complete. 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.411+0000 7fb400dec700 1 Processor -- start 2024-03-18T20:25:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 -- start start 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc19b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19bfd0 con 0x7fb3fc0feed0 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19c140 con 0x7fb3fc106ef0 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb400dec700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19c300 con 0x7fb3fc0ff890 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37054/0 (socket says 172.21.15.98:37054) 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 -- 172.21.15.98:0/3262238703 learned_addr learned my addr 172.21.15.98:0/3262238703 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fad9d700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc19b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3f9d9b700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 -- 172.21.15.98:0/3262238703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 msgr2=0x7fb3fc197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 -- 172.21.15.98:0/3262238703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 msgr2=0x7fb3fc19b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc19b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 -- 172.21.15.98:0/3262238703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0095f0 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3f9d9b700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fad9d700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc19b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.412+0000 7fb3fa59c700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006010 tx=0x7fb3ec009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec01d070 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb3ec00fc80 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc19c580 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec022ce0 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1a3e40 con 0x7fb3fc0feed0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.413+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc109370 con 0x7fb3fc0feed0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.414+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb3ec022470 con 0x7fb3fc0feed0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.415+0000 7fb3eb7fe700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 0x7fb3e40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.415+0000 7fb3f9d9b700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 0x7fb3e40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.416+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb3ec0cdee0 con 0x7fb3fc0feed0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.416+0000 7fb3f9d9b700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 0x7fb3e40a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc198630 tx=0x7fb3f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.419+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3ec097350 con 0x7fb3fc0feed0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.632+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fb3fc04ea80 con 0x7fb3fc0feed0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi098.stdout:292057776150 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.632+0000 7fb3eb7fe700 1 -- 172.21.15.98:0/3262238703 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb3ec09a970 con 0x7fb3fc0feed0 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 msgr2=0x7fb3e40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 0x7fb3e40a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc198630 tx=0x7fb3f0008040 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 msgr2=0x7fb3fc1970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006010 tx=0x7fb3ec009ee0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 shutdown_connections 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fb3e40a11b0 0x7fb3e40a3670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3fc106ef0 0x7fb3fc19b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3fc0feed0 0x7fb3fc1970d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 --2- 172.21.15.98:0/3262238703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3fc0ff890 0x7fb3fc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.634+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 >> 172.21.15.98:0/3262238703 conn(0x7fb3fc075e80 msgr2=0x7fb3fc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.635+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 shutdown_connections 2024-03-18T20:25:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.635+0000 7fb400dec700 1 -- 172.21.15.98:0/3262238703 wait complete. 2024-03-18T20:25:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.811+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2ff804ea80 con 0x7f2ff80699c0 2024-03-18T20:25:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.811+0000 7f2ff4ff9700 1 -- 172.21.15.98:0/2425674615 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2fe809ab60 con 0x7f2ff80699c0 2024-03-18T20:25:30.816 INFO:teuthology.orchestra.run.smithi098.stdout:107374182446 2024-03-18T20:25:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 msgr2=0x7f2fe00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 0x7f2fe00a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2ff819a860 tx=0x7f2fec007400 comp rx=0 tx=0).stop 2024-03-18T20:25:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 msgr2=0x7f2ff8199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006010 tx=0x7f2fe8004730 comp rx=0 tx=0).stop 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 shutdown_connections 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ff810d600 0x7f2ff819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ff80699c0 0x7f2ff8199350 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fe00a11b0 0x7f2fe00a3670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 --2- 172.21.15.98:0/2425674615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ff8101c60 0x7f2ff8199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 >> 172.21.15.98:0/2425674615 conn(0x7f2ff8076b40 msgr2=0x7f2ff80fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 shutdown_connections 2024-03-18T20:25:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.814+0000 7f2ffe0d0700 1 -- 172.21.15.98:0/2425674615 wait complete. 2024-03-18T20:25:30.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:30 smithi098 ceph-mon[43462]: pgmap v247: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:30.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:30 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/501770671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T20:25:30.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:30 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3262238703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T20:25:30.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168603 got 240518168605 for osd.8 2024-03-18T20:25:30.870 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:30.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.868+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f847404ea80 con 0x7f8474100920 2024-03-18T20:25:30.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.868+0000 7f8470ff9700 1 -- 172.21.15.98:0/3875180861 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f8468016090 con 0x7f8474100920 2024-03-18T20:25:30.873 INFO:teuthology.orchestra.run.smithi098.stdout:326417514514 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 msgr2=0x7f845c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 0x7f845c0a3670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f84640059d0 tx=0x7f8464005960 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 msgr2=0x7f84741976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f846800d950 tx=0x7f846800dd10 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 shutdown_connections 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f845c0a11b0 0x7f845c0a3670 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8474104fb0 0x7f847419ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f84740fff70 0x7f8474197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 --2- 172.21.15.98:0/3875180861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8474100920 0x7f84741976c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 >> 172.21.15.98:0/3875180861 conn(0x7f8474075d50 msgr2=0x7f8474068bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 shutdown_connections 2024-03-18T20:25:30.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:30.870+0000 7f8479b8b700 1 -- 172.21.15.98:0/3875180861 wait complete. 2024-03-18T20:25:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:30 smithi136 ceph-mon[48073]: pgmap v247: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:30 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/501770671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T20:25:30.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:30 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3262238703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T20:25:31.087 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574900 got 55834574902 for osd.1 2024-03-18T20:25:31.087 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:30 smithi169 ceph-mon[48529]: pgmap v247: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-18T20:25:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:30 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/501770671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T20:25:31.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:30 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3262238703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T20:25:31.543 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182445 got 107374182446 for osd.3 2024-03-18T20:25:31.543 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:31.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514513 got 326417514514 for osd.11 2024-03-18T20:25:31.544 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:31.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:31 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2425674615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T20:25:31.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:31 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3875180861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T20:25:31.814 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776148 got 292057776150 for osd.10 2024-03-18T20:25:31.814 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:31.815 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T20:25:31.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T20:25:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:31 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2425674615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T20:25:31.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:31 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3875180861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T20:25:32.038 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:32.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:31 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2425674615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T20:25:32.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:31 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3875180861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T20:25:32.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:32 smithi098 ceph-mon[43462]: pgmap v248: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-18T20:25:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:32 smithi136 ceph-mon[48073]: pgmap v248: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-18T20:25:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.004+0000 7fe200421700 1 -- 172.21.15.98:0/4259849521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80ff560 msgr2=0x7fe1f80ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.004+0000 7fe200421700 1 --2- 172.21.15.98:0/4259849521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80ff560 0x7fe1f80ff940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009990 tx=0x7fe1e8009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- 172.21.15.98:0/4259849521 shutdown_connections 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- 172.21.15.98:0/4259849521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f8106130 0x7fe1f810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- 172.21.15.98:0/4259849521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80ff560 0x7fe1f80ff940 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- 172.21.15.98:0/4259849521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- 172.21.15.98:0/4259849521 >> 172.21.15.98:0/4259849521 conn(0x7fe1f80762b0 msgr2=0x7fe1f80fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- 172.21.15.98:0/4259849521 shutdown_connections 2024-03-18T20:25:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- 172.21.15.98:0/4259849521 wait complete. 2024-03-18T20:25:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 Processor -- start 2024-03-18T20:25:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- start start 2024-03-18T20:25:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f80ff560 0x7fe1f81970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f819bfd0 con 0x7fe1f8106130 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f819c140 con 0x7fe1f80ff560 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.005+0000 7fe200421700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f819c300 con 0x7fe1f80fff10 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37128/0 (socket says 172.21.15.98:37128) 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 -- 172.21.15.98:0/2369133538 learned_addr learned my addr 172.21.15.98:0/2369133538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe1bd700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f80ff560 0x7fe1f81970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fd9bc700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 -- 172.21.15.98:0/2369133538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 msgr2=0x7fe1f8197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 -- 172.21.15.98:0/2369133538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f80ff560 msgr2=0x7fe1f81970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f80ff560 0x7fe1f81970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 -- 172.21.15.98:0/2369133538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e80095f0 con 0x7fe1f8106130 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fd9bc700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1fe9be700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0007ee0 tx=0x7fe1f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0017070 con 0x7fe1f8106130 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f819c5e0 con 0x7fe1f8106130 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe1f000f040 con 0x7fe1f8106130 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0005710 con 0x7fe1f8106130 2024-03-18T20:25:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.006+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f81a3e40 con 0x7fe1f8106130 2024-03-18T20:25:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.007+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f81093c0 con 0x7fe1f8106130 2024-03-18T20:25:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.008+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe1f0005870 con 0x7fe1f8106130 2024-03-18T20:25:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.009+0000 7fe1ef7fe700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 0x7fe1e40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.009+0000 7fe1fe1bd700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 0x7fe1e40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.009+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe1f00cc450 con 0x7fe1f8106130 2024-03-18T20:25:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.009+0000 7fe1fe1bd700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 0x7fe1e40a36f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800b5c0 tx=0x7fe1e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.013+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1f0095830 con 0x7fe1f8106130 2024-03-18T20:25:33.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:32 smithi169 ceph-mon[48529]: pgmap v248: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-18T20:25:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.201+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1f81983e0 con 0x7fe1e40a1230 2024-03-18T20:25:33.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.213+0000 7fe1ef7fe700 1 -- 172.21.15.98:0/2369133538 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+421971 (secure 0 0 0) 0x7fe1f81983e0 con 0x7fe1e40a1230 2024-03-18T20:25:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 msgr2=0x7fe1e40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 0x7fe1e40a36f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800b5c0 tx=0x7fe1e80058e0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 msgr2=0x7fe1f819b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0007ee0 tx=0x7fe1f000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 shutdown_connections 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f8106130 0x7fe1f819b9c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe1e40a1230 0x7fe1e40a36f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1f80ff560 0x7fe1f81970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 --2- 172.21.15.98:0/2369133538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1f80fff10 0x7fe1f8197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 >> 172.21.15.98:0/2369133538 conn(0x7fe1f80762b0 msgr2=0x7fe1f80fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.216+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 shutdown_connections 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:33.217+0000 7fe200421700 1 -- 172.21.15.98:0/2369133538 wait complete. 2024-03-18T20:25:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-18T20:25:33.587 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-03-18T20:25:31.583676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655538,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":330,"num_read_kb":278,"num_write":337,"num_write_kb":3051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337296,"kb_used_data":13104,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604528,"statfs":{"total":1151940427776,"available":1151595036672,"internally_reserved":0,"allocated":13418496,"data_stored":7486287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":42,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003373"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430988+0000","last_change":"2024-03-18T20:24:41.360546+0000","last_active":"2024-03-18T20:24:45.430988+0000","last_peered":"2024-03-18T20:24:45.430988+0000","last_clean":"2024-03-18T20:24:45.430988+0000","last_became_active":"2024-03-18T20:24:41.360221+0000","last_became_peered":"2024-03-18T20:24:41.360221+0000","last_unstale":"2024-03-18T20:24:45.430988+0000","last_undegraded":"2024-03-18T20:24:45.430988+0000","last_fullsized":"2024-03-18T20:24:45.430988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:23.329362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441941+0000","last_change":"2024-03-18T20:24:41.354655+0000","last_active":"2024-03-18T20:24:45.441941+0000","last_peered":"2024-03-18T20:24:45.441941+0000","last_clean":"2024-03-18T20:24:45.441941+0000","last_became_active":"2024-03-18T20:24:41.354340+0000","last_became_peered":"2024-03-18T20:24:41.354340+0000","last_unstale":"2024-03-18T20:24:45.441941+0000","last_undegraded":"2024-03-18T20:24:45.441941+0000","last_fullsized":"2024-03-18T20:24:45.441941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:18.785662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440207+0000","last_change":"2024-03-18T20:24:41.351670+0000","last_active":"2024-03-18T20:24:45.440207+0000","last_peered":"2024-03-18T20:24:45.440207+0000","last_clean":"2024-03-18T20:24:45.440207+0000","last_became_active":"2024-03-18T20:24:41.351379+0000","last_became_peered":"2024-03-18T20:24:41.351379+0000","last_unstale":"2024-03-18T20:24:45.440207+0000","last_undegraded":"2024-03-18T20:24:45.440207+0000","last_fullsized":"2024-03-18T20:24:45.440207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:50.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436769+0000","last_change":"2024-03-18T20:24:41.364821+0000","last_active":"2024-03-18T20:24:45.436769+0000","last_peered":"2024-03-18T20:24:45.436769+0000","last_clean":"2024-03-18T20:24:45.436769+0000","last_became_active":"2024-03-18T20:24:41.364496+0000","last_became_peered":"2024-03-18T20:24:41.364496+0000","last_unstale":"2024-03-18T20:24:45.436769+0000","last_undegraded":"2024-03-18T20:24:45.436769+0000","last_fullsized":"2024-03-18T20:24:45.436769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:53.506013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431016+0000","last_change":"2024-03-18T20:24:41.356223+0000","last_active":"2024-03-18T20:24:45.431016+0000","last_peered":"2024-03-18T20:24:45.431016+0000","last_clean":"2024-03-18T20:24:45.431016+0000","last_became_active":"2024-03-18T20:24:41.355831+0000","last_became_peered":"2024-03-18T20:24:41.355831+0000","last_unstale":"2024-03-18T20:24:45.431016+0000","last_undegraded":"2024-03-18T20:24:45.431016+0000","last_fullsized":"2024-03-18T20:24:45.431016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:07.593472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852430+0000","last_change":"2024-03-18T20:24:41.367208+0000","last_active":"2024-03-18T20:24:45.852430+0000","last_peered":"2024-03-18T20:24:45.852430+0000","last_clean":"2024-03-18T20:24:45.852430+0000","last_became_active":"2024-03-18T20:24:41.366885+0000","last_became_peered":"2024-03-18T20:24:41.366885+0000","last_unstale":"2024-03-18T20:24:45.852430+0000","last_undegraded":"2024-03-18T20:24:45.852430+0000","last_fullsized":"2024-03-18T20:24:45.852430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:34.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436116+0000","last_change":"2024-03-18T20:24:41.367935+0000","last_active":"2024-03-18T20:24:45.436116+0000","last_peered":"2024-03-18T20:24:45.436116+0000","last_clean":"2024-03-18T20:24:45.436116+0000","last_became_active":"2024-03-18T20:24:41.367615+0000","last_became_peered":"2024-03-18T20:24:41.367615+0000","last_unstale":"2024-03-18T20:24:45.436116+0000","last_undegraded":"2024-03-18T20:24:45.436116+0000","last_fullsized":"2024-03-18T20:24:45.436116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:41.242857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426070+0000","last_change":"2024-03-18T20:24:41.369045+0000","last_active":"2024-03-18T20:24:45.426070+0000","last_peered":"2024-03-18T20:24:45.426070+0000","last_clean":"2024-03-18T20:24:45.426070+0000","last_became_active":"2024-03-18T20:24:41.368669+0000","last_became_peered":"2024-03-18T20:24:41.368669+0000","last_unstale":"2024-03-18T20:24:45.426070+0000","last_undegraded":"2024-03-18T20:24:45.426070+0000","last_fullsized":"2024-03-18T20:24:45.426070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:09.538243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432055+0000","last_change":"2024-03-18T20:24:41.366520+0000","last_active":"2024-03-18T20:24:45.432055+0000","last_peered":"2024-03-18T20:24:45.432055+0000","last_clean":"2024-03-18T20:24:45.432055+0000","last_became_active":"2024-03-18T20:24:41.366155+0000","last_became_peered":"2024-03-18T20:24:41.366155+0000","last_unstale":"2024-03-18T20:24:45.432055+0000","last_undegraded":"2024-03-18T20:24:45.432055+0000","last_fullsized":"2024-03-18T20:24:45.432055+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:33.151394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852478+0000","last_change":"2024-03-18T20:24:41.370742+0000","last_active":"2024-03-18T20:24:45.852478+0000","last_peered":"2024-03-18T20:24:45.852478+0000","last_clean":"2024-03-18T20:24:45.852478+0000","last_became_active":"2024-03-18T20:24:41.370438+0000","last_became_peered":"2024-03-18T20:24:41.370438+0000","last_unstale":"2024-03-18T20:24:45.852478+0000","last_undegraded":"2024-03-18T20:24:45.852478+0000","last_fullsized":"2024-03-18T20:24:45.852478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:04.331608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436604+0000","last_change":"2024-03-18T20:24:41.366970+0000","last_active":"2024-03-18T20:24:45.436604+0000","last_peered":"2024-03-18T20:24:45.436604+0000","last_clean":"2024-03-18T20:24:45.436604+0000","last_became_active":"2024-03-18T20:24:41.366440+0000","last_became_peered":"2024-03-18T20:24:41.366440+0000","last_unstale":"2024-03-18T20:24:45.436604+0000","last_undegraded":"2024-03-18T20:24:45.436604+0000","last_fullsized":"2024-03-18T20:24:45.436604+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:10.050606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436670+0000","last_change":"2024-03-18T20:24:41.372375+0000","last_active":"2024-03-18T20:24:45.436670+0000","last_peered":"2024-03-18T20:24:45.436670+0000","last_clean":"2024-03-18T20:24:45.436670+0000","last_became_active":"2024-03-18T20:24:41.371028+0000","last_became_peered":"2024-03-18T20:24:41.371028+0000","last_unstale":"2024-03-18T20:24:45.436670+0000","last_undegraded":"2024-03-18T20:24:45.436670+0000","last_fullsized":"2024-03-18T20:24:45.436670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:39.906774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430787+0000","last_change":"2024-03-18T20:24:41.368252+0000","last_active":"2024-03-18T20:24:45.430787+0000","last_peered":"2024-03-18T20:24:45.430787+0000","last_clean":"2024-03-18T20:24:45.430787+0000","last_became_active":"2024-03-18T20:24:41.367911+0000","last_became_peered":"2024-03-18T20:24:41.367911+0000","last_unstale":"2024-03-18T20:24:45.430787+0000","last_undegraded":"2024-03-18T20:24:45.430787+0000","last_fullsized":"2024-03-18T20:24:45.430787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:36.250924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440015+0000","last_change":"2024-03-18T20:24:41.358690+0000","last_active":"2024-03-18T20:24:45.440015+0000","last_peered":"2024-03-18T20:24:45.440015+0000","last_clean":"2024-03-18T20:24:45.440015+0000","last_became_active":"2024-03-18T20:24:41.358254+0000","last_became_peered":"2024-03-18T20:24:41.358254+0000","last_unstale":"2024-03-18T20:24:45.440015+0000","last_undegraded":"2024-03-18T20:24:45.440015+0000","last_fullsized":"2024-03-18T20:24:45.440015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:51.466799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440570+0000","last_change":"2024-03-18T20:24:41.364337+0000","last_active":"2024-03-18T20:24:45.440570+0000","last_peered":"2024-03-18T20:24:45.440570+0000","last_clean":"2024-03-18T20:24:45.440570+0000","last_became_active":"2024-03-18T20:24:41.364026+0000","last_became_peered":"2024-03-18T20:24:41.364026+0000","last_unstale":"2024-03-18T20:24:45.440570+0000","last_undegraded":"2024-03-18T20:24:45.440570+0000","last_fullsized":"2024-03-18T20:24:45.440570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:34.336403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431248+0000","last_change":"2024-03-18T20:24:41.360989+0000","last_active":"2024-03-18T20:24:45.431248+0000","last_peered":"2024-03-18T20:24:45.431248+0000","last_clean":"2024-03-18T20:24:45.431248+0000","last_became_active":"2024-03-18T20:24:41.360097+0000","last_became_peered":"2024-03-18T20:24:41.360097+0000","last_unstale":"2024-03-18T20:24:45.431248+0000","last_undegraded":"2024-03-18T20:24:45.431248+0000","last_fullsized":"2024-03-18T20:24:45.431248+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:18.522013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430879+0000","last_change":"2024-03-18T20:24:41.361221+0000","last_active":"2024-03-18T20:24:45.430879+0000","last_peered":"2024-03-18T20:24:45.430879+0000","last_clean":"2024-03-18T20:24:45.430879+0000","last_became_active":"2024-03-18T20:24:41.360924+0000","last_became_peered":"2024-03-18T20:24:41.360924+0000","last_unstale":"2024-03-18T20:24:45.430879+0000","last_undegraded":"2024-03-18T20:24:45.430879+0000","last_fullsized":"2024-03-18T20:24:45.430879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:24.233761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430265+0000","last_change":"2024-03-18T20:24:41.358145+0000","last_active":"2024-03-18T20:24:45.430265+0000","last_peered":"2024-03-18T20:24:45.430265+0000","last_clean":"2024-03-18T20:24:45.430265+0000","last_became_active":"2024-03-18T20:24:41.357862+0000","last_became_peered":"2024-03-18T20:24:41.357862+0000","last_unstale":"2024-03-18T20:24:45.430265+0000","last_undegraded":"2024-03-18T20:24:45.430265+0000","last_fullsized":"2024-03-18T20:24:45.430265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:01.582221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427416+0000","last_change":"2024-03-18T20:24:41.359214+0000","last_active":"2024-03-18T20:24:45.427416+0000","last_peered":"2024-03-18T20:24:45.427416+0000","last_clean":"2024-03-18T20:24:45.427416+0000","last_became_active":"2024-03-18T20:24:41.357976+0000","last_became_peered":"2024-03-18T20:24:41.357976+0000","last_unstale":"2024-03-18T20:24:45.427416+0000","last_undegraded":"2024-03-18T20:24:45.427416+0000","last_fullsized":"2024-03-18T20:24:45.427416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:36.873383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854870+0000","last_change":"2024-03-18T20:24:45.440427+0000","last_active":"2024-03-18T20:24:45.854870+0000","last_peered":"2024-03-18T20:24:45.854870+0000","last_clean":"2024-03-18T20:24:45.854870+0000","last_became_active":"2024-03-18T20:24:45.440023+0000","last_became_peered":"2024-03-18T20:24:45.440023+0000","last_unstale":"2024-03-18T20:24:45.854870+0000","last_undegraded":"2024-03-18T20:24:45.854870+0000","last_fullsized":"2024-03-18T20:24:45.854870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:37.745854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479395+0000","last_change":"2024-03-18T20:24:40.945829+0000","last_active":"2024-03-18T20:24:59.479395+0000","last_peered":"2024-03-18T20:24:59.479395+0000","last_clean":"2024-03-18T20:24:59.479395+0000","last_became_active":"2024-03-18T20:24:40.945498+0000","last_became_peered":"2024-03-18T20:24:40.945498+0000","last_unstale":"2024-03-18T20:24:59.479395+0000","last_undegraded":"2024-03-18T20:24:59.479395+0000","last_fullsized":"2024-03-18T20:24:59.479395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:40.221678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426446+0000","last_change":"2024-03-18T20:24:41.368911+0000","last_active":"2024-03-18T20:24:45.426446+0000","last_peered":"2024-03-18T20:24:45.426446+0000","last_clean":"2024-03-18T20:24:45.426446+0000","last_became_active":"2024-03-18T20:24:41.368603+0000","last_became_peered":"2024-03-18T20:24:41.368603+0000","last_unstale":"2024-03-18T20:24:45.426446+0000","last_undegraded":"2024-03-18T20:24:45.426446+0000","last_fullsized":"2024-03-18T20:24:45.426446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:37.722935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961438+0000","last_change":"2024-03-18T20:24:45.439904+0000","last_active":"2024-03-18T20:24:45.961438+0000","last_peered":"2024-03-18T20:24:45.961438+0000","last_clean":"2024-03-18T20:24:45.961438+0000","last_became_active":"2024-03-18T20:24:45.439489+0000","last_became_peered":"2024-03-18T20:24:45.439489+0000","last_unstale":"2024-03-18T20:24:45.961438+0000","last_undegraded":"2024-03-18T20:24:45.961438+0000","last_fullsized":"2024-03-18T20:24:45.961438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:21.696148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962086+0000","last_change":"2024-03-18T20:24:40.942150+0000","last_active":"2024-03-18T20:24:45.962086+0000","last_peered":"2024-03-18T20:24:45.962086+0000","last_clean":"2024-03-18T20:24:45.962086+0000","last_became_active":"2024-03-18T20:24:40.941848+0000","last_became_peered":"2024-03-18T20:24:40.941848+0000","last_unstale":"2024-03-18T20:24:45.962086+0000","last_undegraded":"2024-03-18T20:24:45.962086+0000","last_fullsized":"2024-03-18T20:24:45.962086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:44:08.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442127+0000","last_change":"2024-03-18T20:24:41.359112+0000","last_active":"2024-03-18T20:24:45.442127+0000","last_peered":"2024-03-18T20:24:45.442127+0000","last_clean":"2024-03-18T20:24:45.442127+0000","last_became_active":"2024-03-18T20:24:41.358817+0000","last_became_peered":"2024-03-18T20:24:41.358817+0000","last_unstale":"2024-03-18T20:24:45.442127+0000","last_undegraded":"2024-03-18T20:24:45.442127+0000","last_fullsized":"2024-03-18T20:24:45.442127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:25:15.699466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962679+0000","last_change":"2024-03-18T20:24:45.445996+0000","last_active":"2024-03-18T20:24:45.962679+0000","last_peered":"2024-03-18T20:24:45.962679+0000","last_clean":"2024-03-18T20:24:45.962679+0000","last_became_active":"2024-03-18T20:24:45.445552+0000","last_became_peered":"2024-03-18T20:24:45.445552+0000","last_unstale":"2024-03-18T20:24:45.962679+0000","last_undegraded":"2024-03-18T20:24:45.962679+0000","last_fullsized":"2024-03-18T20:24:45.962679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:26.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476209+0000","last_change":"2024-03-18T20:24:40.951499+0000","last_active":"2024-03-18T20:24:59.476209+0000","last_peered":"2024-03-18T20:24:59.476209+0000","last_clean":"2024-03-18T20:24:59.476209+0000","last_became_active":"2024-03-18T20:24:40.951183+0000","last_became_peered":"2024-03-18T20:24:40.951183+0000","last_unstale":"2024-03-18T20:24:59.476209+0000","last_undegraded":"2024-03-18T20:24:59.476209+0000","last_fullsized":"2024-03-18T20:24:59.476209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:25.936370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432254+0000","last_change":"2024-03-18T20:24:41.354835+0000","last_active":"2024-03-18T20:24:45.432254+0000","last_peered":"2024-03-18T20:24:45.432254+0000","last_clean":"2024-03-18T20:24:45.432254+0000","last_became_active":"2024-03-18T20:24:41.354486+0000","last_became_peered":"2024-03-18T20:24:41.354486+0000","last_unstale":"2024-03-18T20:24:45.432254+0000","last_undegraded":"2024-03-18T20:24:45.432254+0000","last_fullsized":"2024-03-18T20:24:45.432254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961610+0000","last_change":"2024-03-18T20:24:45.446772+0000","last_active":"2024-03-18T20:24:45.961610+0000","last_peered":"2024-03-18T20:24:45.961610+0000","last_clean":"2024-03-18T20:24:45.961610+0000","last_became_active":"2024-03-18T20:24:45.446296+0000","last_became_peered":"2024-03-18T20:24:45.446296+0000","last_unstale":"2024-03-18T20:24:45.961610+0000","last_undegraded":"2024-03-18T20:24:45.961610+0000","last_fullsized":"2024-03-18T20:24:45.961610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:41:20.784772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431809+0000","last_change":"2024-03-18T20:24:40.948077+0000","last_active":"2024-03-18T20:24:45.431809+0000","last_peered":"2024-03-18T20:24:45.431809+0000","last_clean":"2024-03-18T20:24:45.431809+0000","last_became_active":"2024-03-18T20:24:40.946970+0000","last_became_peered":"2024-03-18T20:24:40.946970+0000","last_unstale":"2024-03-18T20:24:45.431809+0000","last_undegraded":"2024-03-18T20:24:45.431809+0000","last_fullsized":"2024-03-18T20:24:45.431809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:34.558466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437521+0000","last_change":"2024-03-18T20:24:41.371040+0000","last_active":"2024-03-18T20:24:45.437521+0000","last_peered":"2024-03-18T20:24:45.437521+0000","last_clean":"2024-03-18T20:24:45.437521+0000","last_became_active":"2024-03-18T20:24:41.370654+0000","last_became_peered":"2024-03-18T20:24:41.370654+0000","last_unstale":"2024-03-18T20:24:45.437521+0000","last_undegraded":"2024-03-18T20:24:45.437521+0000","last_fullsized":"2024-03-18T20:24:45.437521+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854817+0000","last_change":"2024-03-18T20:24:45.438825+0000","last_active":"2024-03-18T20:24:45.854817+0000","last_peered":"2024-03-18T20:24:45.854817+0000","last_clean":"2024-03-18T20:24:45.854817+0000","last_became_active":"2024-03-18T20:24:45.438418+0000","last_became_peered":"2024-03-18T20:24:45.438418+0000","last_unstale":"2024-03-18T20:24:45.854817+0000","last_undegraded":"2024-03-18T20:24:45.854817+0000","last_fullsized":"2024-03-18T20:24:45.854817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:52.562628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:28.528718+0000","last_change":"2024-03-18T20:24:40.941503+0000","last_active":"2024-03-18T20:25:28.528718+0000","last_peered":"2024-03-18T20:25:28.528718+0000","last_clean":"2024-03-18T20:25:28.528718+0000","last_became_active":"2024-03-18T20:24:40.941150+0000","last_became_peered":"2024-03-18T20:24:40.941150+0000","last_unstale":"2024-03-18T20:25:28.528718+0000","last_undegraded":"2024-03-18T20:25:28.528718+0000","last_fullsized":"2024-03-18T20:25:28.528718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:02.509071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430556+0000","last_change":"2024-03-18T20:24:41.369688+0000","last_active":"2024-03-18T20:24:45.430556+0000","last_peered":"2024-03-18T20:24:45.430556+0000","last_clean":"2024-03-18T20:24:45.430556+0000","last_became_active":"2024-03-18T20:24:41.369361+0000","last_became_peered":"2024-03-18T20:24:41.369361+0000","last_unstale":"2024-03-18T20:24:45.430556+0000","last_undegraded":"2024-03-18T20:24:45.430556+0000","last_fullsized":"2024-03-18T20:24:45.430556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:23.128929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854843+0000","last_change":"2024-03-18T20:24:45.440139+0000","last_active":"2024-03-18T20:24:45.854843+0000","last_peered":"2024-03-18T20:24:45.854843+0000","last_clean":"2024-03-18T20:24:45.854843+0000","last_became_active":"2024-03-18T20:24:45.439758+0000","last_became_peered":"2024-03-18T20:24:45.439758+0000","last_unstale":"2024-03-18T20:24:45.854843+0000","last_undegraded":"2024-03-18T20:24:45.854843+0000","last_fullsized":"2024-03-18T20:24:45.854843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:57.064268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476530+0000","last_change":"2024-03-18T20:24:40.944915+0000","last_active":"2024-03-18T20:24:59.476530+0000","last_peered":"2024-03-18T20:24:59.476530+0000","last_clean":"2024-03-18T20:24:59.476530+0000","last_became_active":"2024-03-18T20:24:40.944547+0000","last_became_peered":"2024-03-18T20:24:40.944547+0000","last_unstale":"2024-03-18T20:24:59.476530+0000","last_undegraded":"2024-03-18T20:24:59.476530+0000","last_fullsized":"2024-03-18T20:24:59.476530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:36.732318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437751+0000","last_change":"2024-03-18T20:24:41.371113+0000","last_active":"2024-03-18T20:24:45.437751+0000","last_peered":"2024-03-18T20:24:45.437751+0000","last_clean":"2024-03-18T20:24:45.437751+0000","last_became_active":"2024-03-18T20:24:41.370764+0000","last_became_peered":"2024-03-18T20:24:41.370764+0000","last_unstale":"2024-03-18T20:24:45.437751+0000","last_undegraded":"2024-03-18T20:24:45.437751+0000","last_fullsized":"2024-03-18T20:24:45.437751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:51.181585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449570+0000","last_change":"2024-03-18T20:24:45.444798+0000","last_active":"2024-03-18T20:24:45.449570+0000","last_peered":"2024-03-18T20:24:45.449570+0000","last_clean":"2024-03-18T20:24:45.449570+0000","last_became_active":"2024-03-18T20:24:45.444374+0000","last_became_peered":"2024-03-18T20:24:45.444374+0000","last_unstale":"2024-03-18T20:24:45.449570+0000","last_undegraded":"2024-03-18T20:24:45.449570+0000","last_fullsized":"2024-03-18T20:24:45.449570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:54.055384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476781+0000","last_change":"2024-03-18T20:24:40.936710+0000","last_active":"2024-03-18T20:24:59.476781+0000","last_peered":"2024-03-18T20:24:59.476781+0000","last_clean":"2024-03-18T20:24:59.476781+0000","last_became_active":"2024-03-18T20:24:40.936360+0000","last_became_peered":"2024-03-18T20:24:40.936360+0000","last_unstale":"2024-03-18T20:24:59.476781+0000","last_undegraded":"2024-03-18T20:24:59.476781+0000","last_fullsized":"2024-03-18T20:24:59.476781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:08.794520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425954+0000","last_change":"2024-03-18T20:24:41.355109+0000","last_active":"2024-03-18T20:24:45.425954+0000","last_peered":"2024-03-18T20:24:45.425954+0000","last_clean":"2024-03-18T20:24:45.425954+0000","last_became_active":"2024-03-18T20:24:41.354718+0000","last_became_peered":"2024-03-18T20:24:41.354718+0000","last_unstale":"2024-03-18T20:24:45.425954+0000","last_undegraded":"2024-03-18T20:24:45.425954+0000","last_fullsized":"2024-03-18T20:24:45.425954+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:39.147267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963393+0000","last_change":"2024-03-18T20:24:45.451275+0000","last_active":"2024-03-18T20:24:45.963393+0000","last_peered":"2024-03-18T20:24:45.963393+0000","last_clean":"2024-03-18T20:24:45.963393+0000","last_became_active":"2024-03-18T20:24:45.450883+0000","last_became_peered":"2024-03-18T20:24:45.450883+0000","last_unstale":"2024-03-18T20:24:45.963393+0000","last_undegraded":"2024-03-18T20:24:45.963393+0000","last_fullsized":"2024-03-18T20:24:45.963393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:10:41.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961539+0000","last_change":"2024-03-18T20:24:40.939692+0000","last_active":"2024-03-18T20:24:45.961539+0000","last_peered":"2024-03-18T20:24:45.961539+0000","last_clean":"2024-03-18T20:24:45.961539+0000","last_became_active":"2024-03-18T20:24:40.939363+0000","last_became_peered":"2024-03-18T20:24:40.939363+0000","last_unstale":"2024-03-18T20:24:45.961539+0000","last_undegraded":"2024-03-18T20:24:45.961539+0000","last_fullsized":"2024-03-18T20:24:45.961539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:41.445337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430758+0000","last_change":"2024-03-18T20:24:41.357835+0000","last_active":"2024-03-18T20:24:45.430758+0000","last_peered":"2024-03-18T20:24:45.430758+0000","last_clean":"2024-03-18T20:24:45.430758+0000","last_became_active":"2024-03-18T20:24:41.357534+0000","last_became_peered":"2024-03-18T20:24:41.357534+0000","last_unstale":"2024-03-18T20:24:45.430758+0000","last_undegraded":"2024-03-18T20:24:45.430758+0000","last_fullsized":"2024-03-18T20:24:45.430758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:16.662257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440398+0000","last_change":"2024-03-18T20:24:45.438295+0000","last_active":"2024-03-18T20:24:45.440398+0000","last_peered":"2024-03-18T20:24:45.440398+0000","last_clean":"2024-03-18T20:24:45.440398+0000","last_became_active":"2024-03-18T20:24:45.437955+0000","last_became_peered":"2024-03-18T20:24:45.437955+0000","last_unstale":"2024-03-18T20:24:45.440398+0000","last_undegraded":"2024-03-18T20:24:45.440398+0000","last_fullsized":"2024-03-18T20:24:45.440398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:26.435144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:29.529509+0000","last_change":"2024-03-18T20:24:40.945507+0000","last_active":"2024-03-18T20:25:29.529509+0000","last_peered":"2024-03-18T20:25:29.529509+0000","last_clean":"2024-03-18T20:25:29.529509+0000","last_became_active":"2024-03-18T20:24:40.945195+0000","last_became_peered":"2024-03-18T20:24:40.945195+0000","last_unstale":"2024-03-18T20:25:29.529509+0000","last_undegraded":"2024-03-18T20:25:29.529509+0000","last_fullsized":"2024-03-18T20:25:29.529509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:29.173130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431923+0000","last_change":"2024-03-18T20:24:41.356795+0000","last_active":"2024-03-18T20:24:45.431923+0000","last_peered":"2024-03-18T20:24:45.431923+0000","last_clean":"2024-03-18T20:24:45.431923+0000","last_became_active":"2024-03-18T20:24:41.356461+0000","last_became_peered":"2024-03-18T20:24:41.356461+0000","last_unstale":"2024-03-18T20:24:45.431923+0000","last_undegraded":"2024-03-18T20:24:45.431923+0000","last_fullsized":"2024-03-18T20:24:45.431923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:04.230985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444215+0000","last_change":"2024-03-18T20:24:45.439984+0000","last_active":"2024-03-18T20:24:45.444215+0000","last_peered":"2024-03-18T20:24:45.444215+0000","last_clean":"2024-03-18T20:24:45.444215+0000","last_became_active":"2024-03-18T20:24:45.439598+0000","last_became_peered":"2024-03-18T20:24:45.439598+0000","last_unstale":"2024-03-18T20:24:45.444215+0000","last_undegraded":"2024-03-18T20:24:45.444215+0000","last_fullsized":"2024-03-18T20:24:45.444215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:49.481812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469645+0000","last_change":"2024-03-18T20:24:40.943978+0000","last_active":"2024-03-18T20:24:58.469645+0000","last_peered":"2024-03-18T20:24:58.469645+0000","last_clean":"2024-03-18T20:24:58.469645+0000","last_became_active":"2024-03-18T20:24:40.943623+0000","last_became_peered":"2024-03-18T20:24:40.943623+0000","last_unstale":"2024-03-18T20:24:58.469645+0000","last_undegraded":"2024-03-18T20:24:58.469645+0000","last_fullsized":"2024-03-18T20:24:58.469645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:58.995256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442197+0000","last_change":"2024-03-18T20:24:41.365465+0000","last_active":"2024-03-18T20:24:45.442197+0000","last_peered":"2024-03-18T20:24:45.442197+0000","last_clean":"2024-03-18T20:24:45.442197+0000","last_became_active":"2024-03-18T20:24:41.365146+0000","last_became_peered":"2024-03-18T20:24:41.365146+0000","last_unstale":"2024-03-18T20:24:45.442197+0000","last_undegraded":"2024-03-18T20:24:45.442197+0000","last_fullsized":"2024-03-18T20:24:45.442197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:47.903106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962967+0000","last_change":"2024-03-18T20:24:45.440065+0000","last_active":"2024-03-18T20:24:45.962967+0000","last_peered":"2024-03-18T20:24:45.962967+0000","last_clean":"2024-03-18T20:24:45.962967+0000","last_became_active":"2024-03-18T20:24:45.439424+0000","last_became_peered":"2024-03-18T20:24:45.439424+0000","last_unstale":"2024-03-18T20:24:45.962967+0000","last_undegraded":"2024-03-18T20:24:45.962967+0000","last_fullsized":"2024-03-18T20:24:45.962967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:28.350851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471783+0000","last_change":"2024-03-18T20:24:40.942424+0000","last_active":"2024-03-18T20:24:58.471783+0000","last_peered":"2024-03-18T20:24:58.471783+0000","last_clean":"2024-03-18T20:24:58.471783+0000","last_became_active":"2024-03-18T20:24:40.941878+0000","last_became_peered":"2024-03-18T20:24:40.941878+0000","last_unstale":"2024-03-18T20:24:58.471783+0000","last_undegraded":"2024-03-18T20:24:58.471783+0000","last_fullsized":"2024-03-18T20:24:58.471783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:40.746350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432427+0000","last_change":"2024-03-18T20:24:41.372684+0000","last_active":"2024-03-18T20:24:45.432427+0000","last_peered":"2024-03-18T20:24:45.432427+0000","last_clean":"2024-03-18T20:24:45.432427+0000","last_became_active":"2024-03-18T20:24:41.372171+0000","last_became_peered":"2024-03-18T20:24:41.372171+0000","last_unstale":"2024-03-18T20:24:45.432427+0000","last_undegraded":"2024-03-18T20:24:45.432427+0000","last_fullsized":"2024-03-18T20:24:45.432427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:50.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444306+0000","last_change":"2024-03-18T20:24:45.440289+0000","last_active":"2024-03-18T20:24:45.444306+0000","last_peered":"2024-03-18T20:24:45.444306+0000","last_clean":"2024-03-18T20:24:45.444306+0000","last_became_active":"2024-03-18T20:24:45.439910+0000","last_became_peered":"2024-03-18T20:24:45.439910+0000","last_unstale":"2024-03-18T20:24:45.444306+0000","last_undegraded":"2024-03-18T20:24:45.444306+0000","last_fullsized":"2024-03-18T20:24:45.444306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:19.420493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'32","reported_seq":117,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:30.536381+0000","last_change":"2024-03-18T20:24:40.943843+0000","last_active":"2024-03-18T20:25:30.536381+0000","last_peered":"2024-03-18T20:25:30.536381+0000","last_clean":"2024-03-18T20:25:30.536381+0000","last_became_active":"2024-03-18T20:24:40.943521+0000","last_became_peered":"2024-03-18T20:24:40.943521+0000","last_unstale":"2024-03-18T20:25:30.536381+0000","last_undegraded":"2024-03-18T20:25:30.536381+0000","last_fullsized":"2024-03-18T20:25:30.536381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:44.872498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61030,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":69,"num_read_kb":92,"num_write":33,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433854+0000","last_change":"2024-03-18T20:24:41.358139+0000","last_active":"2024-03-18T20:24:45.433854+0000","last_peered":"2024-03-18T20:24:45.433854+0000","last_clean":"2024-03-18T20:24:45.433854+0000","last_became_active":"2024-03-18T20:24:41.357789+0000","last_became_peered":"2024-03-18T20:24:41.357789+0000","last_unstale":"2024-03-18T20:24:45.433854+0000","last_undegraded":"2024-03-18T20:24:45.433854+0000","last_fullsized":"2024-03-18T20:24:45.433854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:49.643210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449427+0000","last_change":"2024-03-18T20:24:45.448899+0000","last_active":"2024-03-18T20:24:45.449427+0000","last_peered":"2024-03-18T20:24:45.449427+0000","last_clean":"2024-03-18T20:24:45.449427+0000","last_became_active":"2024-03-18T20:24:45.448519+0000","last_became_peered":"2024-03-18T20:24:45.448519+0000","last_unstale":"2024-03-18T20:24:45.449427+0000","last_undegraded":"2024-03-18T20:24:45.449427+0000","last_fullsized":"2024-03-18T20:24:45.449427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:54.071495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476499+0000","last_change":"2024-03-18T20:24:40.944455+0000","last_active":"2024-03-18T20:24:59.476499+0000","last_peered":"2024-03-18T20:24:59.476499+0000","last_clean":"2024-03-18T20:24:59.476499+0000","last_became_active":"2024-03-18T20:24:40.944115+0000","last_became_peered":"2024-03-18T20:24:40.944115+0000","last_unstale":"2024-03-18T20:24:59.476499+0000","last_undegraded":"2024-03-18T20:24:59.476499+0000","last_fullsized":"2024-03-18T20:24:59.476499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:06.160375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430066+0000","last_change":"2024-03-18T20:24:41.356074+0000","last_active":"2024-03-18T20:24:45.430066+0000","last_peered":"2024-03-18T20:24:45.430066+0000","last_clean":"2024-03-18T20:24:45.430066+0000","last_became_active":"2024-03-18T20:24:41.355755+0000","last_became_peered":"2024-03-18T20:24:41.355755+0000","last_unstale":"2024-03-18T20:24:45.430066+0000","last_undegraded":"2024-03-18T20:24:45.430066+0000","last_fullsized":"2024-03-18T20:24:45.430066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:54.007791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.443111+0000","last_change":"2024-03-18T20:24:45.440797+0000","last_active":"2024-03-18T20:24:45.443111+0000","last_peered":"2024-03-18T20:24:45.443111+0000","last_clean":"2024-03-18T20:24:45.443111+0000","last_became_active":"2024-03-18T20:24:45.440410+0000","last_became_peered":"2024-03-18T20:24:45.440410+0000","last_unstale":"2024-03-18T20:24:45.443111+0000","last_undegraded":"2024-03-18T20:24:45.443111+0000","last_fullsized":"2024-03-18T20:24:45.443111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:02.586794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471938+0000","last_change":"2024-03-18T20:24:40.944239+0000","last_active":"2024-03-18T20:24:58.471938+0000","last_peered":"2024-03-18T20:24:58.471938+0000","last_clean":"2024-03-18T20:24:58.471938+0000","last_became_active":"2024-03-18T20:24:40.943845+0000","last_became_peered":"2024-03-18T20:24:40.943845+0000","last_unstale":"2024-03-18T20:24:58.471938+0000","last_undegraded":"2024-03-18T20:24:58.471938+0000","last_fullsized":"2024-03-18T20:24:58.471938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:30.966789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436700+0000","last_change":"2024-03-18T20:24:41.373565+0000","last_active":"2024-03-18T20:24:45.436700+0000","last_peered":"2024-03-18T20:24:45.436700+0000","last_clean":"2024-03-18T20:24:45.436700+0000","last_became_active":"2024-03-18T20:24:41.373201+0000","last_became_peered":"2024-03-18T20:24:41.373201+0000","last_unstale":"2024-03-18T20:24:45.436700+0000","last_undegraded":"2024-03-18T20:24:45.436700+0000","last_fullsized":"2024-03-18T20:24:45.436700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:13.118518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.447664+0000","last_change":"2024-03-18T20:24:45.447115+0000","last_active":"2024-03-18T20:24:45.447664+0000","last_peered":"2024-03-18T20:24:45.447664+0000","last_clean":"2024-03-18T20:24:45.447664+0000","last_became_active":"2024-03-18T20:24:45.446316+0000","last_became_peered":"2024-03-18T20:24:45.446316+0000","last_unstale":"2024-03-18T20:24:45.447664+0000","last_undegraded":"2024-03-18T20:24:45.447664+0000","last_fullsized":"2024-03-18T20:24:45.447664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:17.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:26.525457+0000","last_change":"2024-03-18T20:24:40.938678+0000","last_active":"2024-03-18T20:25:26.525457+0000","last_peered":"2024-03-18T20:25:26.525457+0000","last_clean":"2024-03-18T20:25:26.525457+0000","last_became_active":"2024-03-18T20:24:40.938300+0000","last_became_peered":"2024-03-18T20:24:40.938300+0000","last_unstale":"2024-03-18T20:25:26.525457+0000","last_undegraded":"2024-03-18T20:25:26.525457+0000","last_fullsized":"2024-03-18T20:25:26.525457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:21.806721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434871+0000","last_change":"2024-03-18T20:24:41.368977+0000","last_active":"2024-03-18T20:24:45.434871+0000","last_peered":"2024-03-18T20:24:45.434871+0000","last_clean":"2024-03-18T20:24:45.434871+0000","last_became_active":"2024-03-18T20:24:41.368545+0000","last_became_peered":"2024-03-18T20:24:41.368545+0000","last_unstale":"2024-03-18T20:24:45.434871+0000","last_undegraded":"2024-03-18T20:24:45.434871+0000","last_fullsized":"2024-03-18T20:24:45.434871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:23.827215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962771+0000","last_change":"2024-03-18T20:24:45.445759+0000","last_active":"2024-03-18T20:24:45.962771+0000","last_peered":"2024-03-18T20:24:45.962771+0000","last_clean":"2024-03-18T20:24:45.962771+0000","last_became_active":"2024-03-18T20:24:45.445366+0000","last_became_peered":"2024-03-18T20:24:45.445366+0000","last_unstale":"2024-03-18T20:24:45.962771+0000","last_undegraded":"2024-03-18T20:24:45.962771+0000","last_fullsized":"2024-03-18T20:24:45.962771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:59.103814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479110+0000","last_change":"2024-03-18T20:24:40.950980+0000","last_active":"2024-03-18T20:24:59.479110+0000","last_peered":"2024-03-18T20:24:59.479110+0000","last_clean":"2024-03-18T20:24:59.479110+0000","last_became_active":"2024-03-18T20:24:40.950280+0000","last_became_peered":"2024-03-18T20:24:40.950280+0000","last_unstale":"2024-03-18T20:24:59.479110+0000","last_undegraded":"2024-03-18T20:24:59.479110+0000","last_fullsized":"2024-03-18T20:24:59.479110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:16:08.679066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437457+0000","last_change":"2024-03-18T20:24:41.374302+0000","last_active":"2024-03-18T20:24:45.437457+0000","last_peered":"2024-03-18T20:24:45.437457+0000","last_clean":"2024-03-18T20:24:45.437457+0000","last_became_active":"2024-03-18T20:24:41.373974+0000","last_became_peered":"2024-03-18T20:24:41.373974+0000","last_unstale":"2024-03-18T20:24:45.437457+0000","last_undegraded":"2024-03-18T20:24:45.437457+0000","last_fullsized":"2024-03-18T20:24:45.437457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445064+0000","last_change":"2024-03-18T20:24:45.443527+0000","last_active":"2024-03-18T20:24:45.445064+0000","last_peered":"2024-03-18T20:24:45.445064+0000","last_clean":"2024-03-18T20:24:45.445064+0000","last_became_active":"2024-03-18T20:24:45.443105+0000","last_became_peered":"2024-03-18T20:24:45.443105+0000","last_unstale":"2024-03-18T20:24:45.445064+0000","last_undegraded":"2024-03-18T20:24:45.445064+0000","last_fullsized":"2024-03-18T20:24:45.445064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:04.568061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.485093+0000","last_change":"2024-03-18T20:24:40.948223+0000","last_active":"2024-03-18T20:25:00.485093+0000","last_peered":"2024-03-18T20:25:00.485093+0000","last_clean":"2024-03-18T20:25:00.485093+0000","last_became_active":"2024-03-18T20:24:40.947889+0000","last_became_peered":"2024-03-18T20:24:40.947889+0000","last_unstale":"2024-03-18T20:25:00.485093+0000","last_undegraded":"2024-03-18T20:25:00.485093+0000","last_fullsized":"2024-03-18T20:25:00.485093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:33.804508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430232+0000","last_change":"2024-03-18T20:24:41.369441+0000","last_active":"2024-03-18T20:24:45.430232+0000","last_peered":"2024-03-18T20:24:45.430232+0000","last_clean":"2024-03-18T20:24:45.430232+0000","last_became_active":"2024-03-18T20:24:41.368271+0000","last_became_peered":"2024-03-18T20:24:41.368271+0000","last_unstale":"2024-03-18T20:24:45.430232+0000","last_undegraded":"2024-03-18T20:24:45.430232+0000","last_fullsized":"2024-03-18T20:24:45.430232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:20.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962427+0000","last_change":"2024-03-18T20:24:45.441117+0000","last_active":"2024-03-18T20:24:45.962427+0000","last_peered":"2024-03-18T20:24:45.962427+0000","last_clean":"2024-03-18T20:24:45.962427+0000","last_became_active":"2024-03-18T20:24:45.440774+0000","last_became_peered":"2024-03-18T20:24:45.440774+0000","last_unstale":"2024-03-18T20:24:45.962427+0000","last_undegraded":"2024-03-18T20:24:45.962427+0000","last_fullsized":"2024-03-18T20:24:45.962427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:27.034432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431547+0000","last_change":"2024-03-18T20:24:40.940987+0000","last_active":"2024-03-18T20:24:45.431547+0000","last_peered":"2024-03-18T20:24:45.431547+0000","last_clean":"2024-03-18T20:24:45.431547+0000","last_became_active":"2024-03-18T20:24:40.940652+0000","last_became_peered":"2024-03-18T20:24:40.940652+0000","last_unstale":"2024-03-18T20:24:45.431547+0000","last_undegraded":"2024-03-18T20:24:45.431547+0000","last_fullsized":"2024-03-18T20:24:45.431547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:57.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852466+0000","last_change":"2024-03-18T20:24:41.356381+0000","last_active":"2024-03-18T20:24:45.852466+0000","last_peered":"2024-03-18T20:24:45.852466+0000","last_clean":"2024-03-18T20:24:45.852466+0000","last_became_active":"2024-03-18T20:24:41.355078+0000","last_became_peered":"2024-03-18T20:24:41.355078+0000","last_unstale":"2024-03-18T20:24:45.852466+0000","last_undegraded":"2024-03-18T20:24:45.852466+0000","last_fullsized":"2024-03-18T20:24:45.852466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:03.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961701+0000","last_change":"2024-03-18T20:24:45.448305+0000","last_active":"2024-03-18T20:24:45.961701+0000","last_peered":"2024-03-18T20:24:45.961701+0000","last_clean":"2024-03-18T20:24:45.961701+0000","last_became_active":"2024-03-18T20:24:45.447706+0000","last_became_peered":"2024-03-18T20:24:45.447706+0000","last_unstale":"2024-03-18T20:24:45.961701+0000","last_undegraded":"2024-03-18T20:24:45.961701+0000","last_fullsized":"2024-03-18T20:24:45.961701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:22.305139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963464+0000","last_change":"2024-03-18T20:24:40.939482+0000","last_active":"2024-03-18T20:24:45.963464+0000","last_peered":"2024-03-18T20:24:45.963464+0000","last_clean":"2024-03-18T20:24:45.963464+0000","last_became_active":"2024-03-18T20:24:40.938983+0000","last_became_peered":"2024-03-18T20:24:40.938983+0000","last_unstale":"2024-03-18T20:24:45.963464+0000","last_undegraded":"2024-03-18T20:24:45.963464+0000","last_fullsized":"2024-03-18T20:24:45.963464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:00.111633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852446+0000","last_change":"2024-03-18T20:24:41.366749+0000","last_active":"2024-03-18T20:24:45.852446+0000","last_peered":"2024-03-18T20:24:45.852446+0000","last_clean":"2024-03-18T20:24:45.852446+0000","last_became_active":"2024-03-18T20:24:41.366335+0000","last_became_peered":"2024-03-18T20:24:41.366335+0000","last_unstale":"2024-03-18T20:24:45.852446+0000","last_undegraded":"2024-03-18T20:24:45.852446+0000","last_fullsized":"2024-03-18T20:24:45.852446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:36.927348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962802+0000","last_change":"2024-03-18T20:24:45.447585+0000","last_active":"2024-03-18T20:24:45.962802+0000","last_peered":"2024-03-18T20:24:45.962802+0000","last_clean":"2024-03-18T20:24:45.962802+0000","last_became_active":"2024-03-18T20:24:45.447165+0000","last_became_peered":"2024-03-18T20:24:45.447165+0000","last_unstale":"2024-03-18T20:24:45.962802+0000","last_undegraded":"2024-03-18T20:24:45.962802+0000","last_fullsized":"2024-03-18T20:24:45.962802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:14:34.960675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438282+0000","last_change":"2024-03-18T20:24:41.371969+0000","last_active":"2024-03-18T20:24:45.438282+0000","last_peered":"2024-03-18T20:24:45.438282+0000","last_clean":"2024-03-18T20:24:45.438282+0000","last_became_active":"2024-03-18T20:24:41.371516+0000","last_became_peered":"2024-03-18T20:24:41.371516+0000","last_unstale":"2024-03-18T20:24:45.438282+0000","last_undegraded":"2024-03-18T20:24:45.438282+0000","last_fullsized":"2024-03-18T20:24:45.438282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:46.761356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.522980+0000","last_change":"2024-03-18T20:24:40.940375+0000","last_active":"2024-03-18T20:25:25.522980+0000","last_peered":"2024-03-18T20:25:25.522980+0000","last_clean":"2024-03-18T20:25:25.522980+0000","last_became_active":"2024-03-18T20:24:40.940053+0000","last_became_peered":"2024-03-18T20:24:40.940053+0000","last_unstale":"2024-03-18T20:25:25.522980+0000","last_undegraded":"2024-03-18T20:25:25.522980+0000","last_fullsized":"2024-03-18T20:25:25.522980+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:20:17.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962803+0000","last_change":"2024-03-18T20:24:45.443315+0000","last_active":"2024-03-18T20:24:45.962803+0000","last_peered":"2024-03-18T20:24:45.962803+0000","last_clean":"2024-03-18T20:24:45.962803+0000","last_became_active":"2024-03-18T20:24:45.442889+0000","last_became_peered":"2024-03-18T20:24:45.442889+0000","last_unstale":"2024-03-18T20:24:45.962803+0000","last_undegraded":"2024-03-18T20:24:45.962803+0000","last_fullsized":"2024-03-18T20:24:45.962803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:40.097397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425997+0000","last_change":"2024-03-18T20:24:41.368952+0000","last_active":"2024-03-18T20:24:45.425997+0000","last_peered":"2024-03-18T20:24:45.425997+0000","last_clean":"2024-03-18T20:24:45.425997+0000","last_became_active":"2024-03-18T20:24:41.368629+0000","last_became_peered":"2024-03-18T20:24:41.368629+0000","last_unstale":"2024-03-18T20:24:45.425997+0000","last_undegraded":"2024-03-18T20:24:45.425997+0000","last_fullsized":"2024-03-18T20:24:45.425997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:28:58.521712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.483637+0000","last_change":"2024-03-18T20:24:40.938841+0000","last_active":"2024-03-18T20:25:00.483637+0000","last_peered":"2024-03-18T20:25:00.483637+0000","last_clean":"2024-03-18T20:25:00.483637+0000","last_became_active":"2024-03-18T20:24:40.938464+0000","last_became_peered":"2024-03-18T20:24:40.938464+0000","last_unstale":"2024-03-18T20:25:00.483637+0000","last_undegraded":"2024-03-18T20:25:00.483637+0000","last_fullsized":"2024-03-18T20:25:00.483637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:04.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":326,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962726+0000","last_change":"2024-03-18T20:23:54.597665+0000","last_active":"2024-03-18T20:24:45.962726+0000","last_peered":"2024-03-18T20:24:45.962726+0000","last_clean":"2024-03-18T20:24:45.962726+0000","last_became_active":"2024-03-18T20:23:54.597135+0000","last_became_peered":"2024-03-18T20:23:54.597135+0000","last_unstale":"2024-03-18T20:24:45.962726+0000","last_undegraded":"2024-03-18T20:24:45.962726+0000","last_fullsized":"2024-03-18T20:24:45.962726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_clean_scrub_stamp":"2024-03-18T20:21:34.333460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:59.546138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449625+0000","last_change":"2024-03-18T20:24:45.448945+0000","last_active":"2024-03-18T20:24:45.449625+0000","last_peered":"2024-03-18T20:24:45.449625+0000","last_clean":"2024-03-18T20:24:45.449625+0000","last_became_active":"2024-03-18T20:24:45.448585+0000","last_became_peered":"2024-03-18T20:24:45.448585+0000","last_unstale":"2024-03-18T20:24:45.449625+0000","last_undegraded":"2024-03-18T20:24:45.449625+0000","last_fullsized":"2024-03-18T20:24:45.449625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:49.668118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429530+0000","last_change":"2024-03-18T20:24:41.364453+0000","last_active":"2024-03-18T20:24:45.429530+0000","last_peered":"2024-03-18T20:24:45.429530+0000","last_clean":"2024-03-18T20:24:45.429530+0000","last_became_active":"2024-03-18T20:24:41.364047+0000","last_became_peered":"2024-03-18T20:24:41.364047+0000","last_unstale":"2024-03-18T20:24:45.429530+0000","last_undegraded":"2024-03-18T20:24:45.429530+0000","last_fullsized":"2024-03-18T20:24:45.429530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:32.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476629+0000","last_change":"2024-03-18T20:24:40.939926+0000","last_active":"2024-03-18T20:24:59.476629+0000","last_peered":"2024-03-18T20:24:59.476629+0000","last_clean":"2024-03-18T20:24:59.476629+0000","last_became_active":"2024-03-18T20:24:40.939514+0000","last_became_peered":"2024-03-18T20:24:40.939514+0000","last_unstale":"2024-03-18T20:24:59.476629+0000","last_undegraded":"2024-03-18T20:24:59.476629+0000","last_fullsized":"2024-03-18T20:24:59.476629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:41.852567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442992+0000","last_change":"2024-03-18T20:24:45.439880+0000","last_active":"2024-03-18T20:24:45.442992+0000","last_peered":"2024-03-18T20:24:45.442992+0000","last_clean":"2024-03-18T20:24:45.442992+0000","last_became_active":"2024-03-18T20:24:45.439455+0000","last_became_peered":"2024-03-18T20:24:45.439455+0000","last_unstale":"2024-03-18T20:24:45.442992+0000","last_undegraded":"2024-03-18T20:24:45.442992+0000","last_fullsized":"2024-03-18T20:24:45.442992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:26.050401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470311+0000","last_change":"2024-03-18T20:24:40.947764+0000","last_active":"2024-03-18T20:24:58.470311+0000","last_peered":"2024-03-18T20:24:58.470311+0000","last_clean":"2024-03-18T20:24:58.470311+0000","last_became_active":"2024-03-18T20:24:40.946947+0000","last_became_peered":"2024-03-18T20:24:40.946947+0000","last_unstale":"2024-03-18T20:24:58.470311+0000","last_undegraded":"2024-03-18T20:24:58.470311+0000","last_fullsized":"2024-03-18T20:24:58.470311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:38.279134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431036+0000","last_change":"2024-03-18T20:24:41.352213+0000","last_active":"2024-03-18T20:24:45.431036+0000","last_peered":"2024-03-18T20:24:45.431036+0000","last_clean":"2024-03-18T20:24:45.431036+0000","last_became_active":"2024-03-18T20:24:41.351895+0000","last_became_peered":"2024-03-18T20:24:41.351895+0000","last_unstale":"2024-03-18T20:24:45.431036+0000","last_undegraded":"2024-03-18T20:24:45.431036+0000","last_fullsized":"2024-03-18T20:24:45.431036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:55.756782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440641+0000","last_change":"2024-03-18T20:24:45.439483+0000","last_active":"2024-03-18T20:24:45.440641+0000","last_peered":"2024-03-18T20:24:45.440641+0000","last_clean":"2024-03-18T20:24:45.440641+0000","last_became_active":"2024-03-18T20:24:45.436952+0000","last_became_peered":"2024-03-18T20:24:45.436952+0000","last_unstale":"2024-03-18T20:24:45.440641+0000","last_undegraded":"2024-03-18T20:24:45.440641+0000","last_fullsized":"2024-03-18T20:24:45.440641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:33.839948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:05.688972+0000","last_change":"2024-03-18T20:24:40.948184+0000","last_active":"2024-03-18T20:25:05.688972+0000","last_peered":"2024-03-18T20:25:05.688972+0000","last_clean":"2024-03-18T20:25:05.688972+0000","last_became_active":"2024-03-18T20:24:40.947953+0000","last_became_peered":"2024-03-18T20:24:40.947953+0000","last_unstale":"2024-03-18T20:25:05.688972+0000","last_undegraded":"2024-03-18T20:25:05.688972+0000","last_fullsized":"2024-03-18T20:25:05.688972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:51.701653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426390+0000","last_change":"2024-03-18T20:24:41.356563+0000","last_active":"2024-03-18T20:24:45.426390+0000","last_peered":"2024-03-18T20:24:45.426390+0000","last_clean":"2024-03-18T20:24:45.426390+0000","last_became_active":"2024-03-18T20:24:41.356229+0000","last_became_peered":"2024-03-18T20:24:41.356229+0000","last_unstale":"2024-03-18T20:24:45.426390+0000","last_undegraded":"2024-03-18T20:24:45.426390+0000","last_fullsized":"2024-03-18T20:24:45.426390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:20.296131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854824+0000","last_change":"2024-03-18T20:24:45.440937+0000","last_active":"2024-03-18T20:24:45.854824+0000","last_peered":"2024-03-18T20:24:45.854824+0000","last_clean":"2024-03-18T20:24:45.854824+0000","last_became_active":"2024-03-18T20:24:45.440624+0000","last_became_peered":"2024-03-18T20:24:45.440624+0000","last_unstale":"2024-03-18T20:24:45.854824+0000","last_undegraded":"2024-03-18T20:24:45.854824+0000","last_fullsized":"2024-03-18T20:24:45.854824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:41.588153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470417+0000","last_change":"2024-03-18T20:24:40.940622+0000","last_active":"2024-03-18T20:24:58.470417+0000","last_peered":"2024-03-18T20:24:58.470417+0000","last_clean":"2024-03-18T20:24:58.470417+0000","last_became_active":"2024-03-18T20:24:40.940285+0000","last_became_peered":"2024-03-18T20:24:40.940285+0000","last_unstale":"2024-03-18T20:24:58.470417+0000","last_undegraded":"2024-03-18T20:24:58.470417+0000","last_fullsized":"2024-03-18T20:24:58.470417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:03.933412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442035+0000","last_change":"2024-03-18T20:24:41.370212+0000","last_active":"2024-03-18T20:24:45.442035+0000","last_peered":"2024-03-18T20:24:45.442035+0000","last_clean":"2024-03-18T20:24:45.442035+0000","last_became_active":"2024-03-18T20:24:41.369904+0000","last_became_peered":"2024-03-18T20:24:41.369904+0000","last_unstale":"2024-03-18T20:24:45.442035+0000","last_undegraded":"2024-03-18T20:24:45.442035+0000","last_fullsized":"2024-03-18T20:24:45.442035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:13:12.982929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961796+0000","last_change":"2024-03-18T20:24:45.446935+0000","last_active":"2024-03-18T20:24:45.961796+0000","last_peered":"2024-03-18T20:24:45.961796+0000","last_clean":"2024-03-18T20:24:45.961796+0000","last_became_active":"2024-03-18T20:24:45.445971+0000","last_became_peered":"2024-03-18T20:24:45.445971+0000","last_unstale":"2024-03-18T20:24:45.961796+0000","last_undegraded":"2024-03-18T20:24:45.961796+0000","last_fullsized":"2024-03-18T20:24:45.961796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:51.292743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470479+0000","last_change":"2024-03-18T20:24:40.937625+0000","last_active":"2024-03-18T20:24:58.470479+0000","last_peered":"2024-03-18T20:24:58.470479+0000","last_clean":"2024-03-18T20:24:58.470479+0000","last_became_active":"2024-03-18T20:24:40.937275+0000","last_became_peered":"2024-03-18T20:24:40.937275+0000","last_unstale":"2024-03-18T20:24:58.470479+0000","last_undegraded":"2024-03-18T20:24:58.470479+0000","last_fullsized":"2024-03-18T20:24:58.470479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:55.747307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430187+0000","last_change":"2024-03-18T20:24:41.349528+0000","last_active":"2024-03-18T20:24:45.430187+0000","last_peered":"2024-03-18T20:24:45.430187+0000","last_clean":"2024-03-18T20:24:45.430187+0000","last_became_active":"2024-03-18T20:24:41.349113+0000","last_became_peered":"2024-03-18T20:24:41.349113+0000","last_unstale":"2024-03-18T20:24:45.430187+0000","last_undegraded":"2024-03-18T20:24:45.430187+0000","last_fullsized":"2024-03-18T20:24:45.430187+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:06.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444468+0000","last_change":"2024-03-18T20:24:45.441827+0000","last_active":"2024-03-18T20:24:45.444468+0000","last_peered":"2024-03-18T20:24:45.444468+0000","last_clean":"2024-03-18T20:24:45.444468+0000","last_became_active":"2024-03-18T20:24:45.441403+0000","last_became_peered":"2024-03-18T20:24:45.441403+0000","last_unstale":"2024-03-18T20:24:45.444468+0000","last_undegraded":"2024-03-18T20:24:45.444468+0000","last_fullsized":"2024-03-18T20:24:45.444468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:42.530843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470136+0000","last_change":"2024-03-18T20:24:40.943702+0000","last_active":"2024-03-18T20:24:58.470136+0000","last_peered":"2024-03-18T20:24:58.470136+0000","last_clean":"2024-03-18T20:24:58.470136+0000","last_became_active":"2024-03-18T20:24:40.943380+0000","last_became_peered":"2024-03-18T20:24:40.943380+0000","last_unstale":"2024-03-18T20:24:58.470136+0000","last_undegraded":"2024-03-18T20:24:58.470136+0000","last_fullsized":"2024-03-18T20:24:58.470136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:04.567157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431139+0000","last_change":"2024-03-18T20:24:41.352181+0000","last_active":"2024-03-18T20:24:45.431139+0000","last_peered":"2024-03-18T20:24:45.431139+0000","last_clean":"2024-03-18T20:24:45.431139+0000","last_became_active":"2024-03-18T20:24:41.351694+0000","last_became_peered":"2024-03-18T20:24:41.351694+0000","last_unstale":"2024-03-18T20:24:45.431139+0000","last_undegraded":"2024-03-18T20:24:45.431139+0000","last_fullsized":"2024-03-18T20:24:45.431139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:00.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:47.625415+0000","last_change":"2024-03-18T20:24:45.444567+0000","last_active":"2024-03-18T20:24:47.625415+0000","last_peered":"2024-03-18T20:24:47.625415+0000","last_clean":"2024-03-18T20:24:47.625415+0000","last_became_active":"2024-03-18T20:24:45.443996+0000","last_became_peered":"2024-03-18T20:24:45.443996+0000","last_unstale":"2024-03-18T20:24:47.625415+0000","last_undegraded":"2024-03-18T20:24:47.625415+0000","last_fullsized":"2024-03-18T20:24:47.625415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:16.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854563+0000","last_change":"2024-03-18T20:24:40.948872+0000","last_active":"2024-03-18T20:24:45.854563+0000","last_peered":"2024-03-18T20:24:45.854563+0000","last_clean":"2024-03-18T20:24:45.854563+0000","last_became_active":"2024-03-18T20:24:40.948540+0000","last_became_peered":"2024-03-18T20:24:40.948540+0000","last_unstale":"2024-03-18T20:24:45.854563+0000","last_undegraded":"2024-03-18T20:24:45.854563+0000","last_fullsized":"2024-03-18T20:24:45.854563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:57.954340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426231+0000","last_change":"2024-03-18T20:24:41.355552+0000","last_active":"2024-03-18T20:24:45.426231+0000","last_peered":"2024-03-18T20:24:45.426231+0000","last_clean":"2024-03-18T20:24:45.426231+0000","last_became_active":"2024-03-18T20:24:41.355245+0000","last_became_peered":"2024-03-18T20:24:41.355245+0000","last_unstale":"2024-03-18T20:24:45.426231+0000","last_undegraded":"2024-03-18T20:24:45.426231+0000","last_fullsized":"2024-03-18T20:24:45.426231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:25.833815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961749+0000","last_change":"2024-03-18T20:24:45.443078+0000","last_active":"2024-03-18T20:24:45.961749+0000","last_peered":"2024-03-18T20:24:45.961749+0000","last_clean":"2024-03-18T20:24:45.961749+0000","last_became_active":"2024-03-18T20:24:45.442675+0000","last_became_peered":"2024-03-18T20:24:45.442675+0000","last_unstale":"2024-03-18T20:24:45.961749+0000","last_undegraded":"2024-03-18T20:24:45.961749+0000","last_fullsized":"2024-03-18T20:24:45.961749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:32.262956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472041+0000","last_change":"2024-03-18T20:24:40.943239+0000","last_active":"2024-03-18T20:24:58.472041+0000","last_peered":"2024-03-18T20:24:58.472041+0000","last_clean":"2024-03-18T20:24:58.472041+0000","last_became_active":"2024-03-18T20:24:40.942862+0000","last_became_peered":"2024-03-18T20:24:40.942862+0000","last_unstale":"2024-03-18T20:24:58.472041+0000","last_undegraded":"2024-03-18T20:24:58.472041+0000","last_fullsized":"2024-03-18T20:24:58.472041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:42.560340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437303+0000","last_change":"2024-03-18T20:24:41.361986+0000","last_active":"2024-03-18T20:24:45.437303+0000","last_peered":"2024-03-18T20:24:45.437303+0000","last_clean":"2024-03-18T20:24:45.437303+0000","last_became_active":"2024-03-18T20:24:41.361608+0000","last_became_peered":"2024-03-18T20:24:41.361608+0000","last_unstale":"2024-03-18T20:24:45.437303+0000","last_undegraded":"2024-03-18T20:24:45.437303+0000","last_fullsized":"2024-03-18T20:24:45.437303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:44.540156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441527+0000","last_change":"2024-03-18T20:24:45.440250+0000","last_active":"2024-03-18T20:24:45.441527+0000","last_peered":"2024-03-18T20:24:45.441527+0000","last_clean":"2024-03-18T20:24:45.441527+0000","last_became_active":"2024-03-18T20:24:45.439782+0000","last_became_peered":"2024-03-18T20:24:45.439782+0000","last_unstale":"2024-03-18T20:24:45.441527+0000","last_undegraded":"2024-03-18T20:24:45.441527+0000","last_fullsized":"2024-03-18T20:24:45.441527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:58.415354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476878+0000","last_change":"2024-03-18T20:24:40.945951+0000","last_active":"2024-03-18T20:24:59.476878+0000","last_peered":"2024-03-18T20:24:59.476878+0000","last_clean":"2024-03-18T20:24:59.476878+0000","last_became_active":"2024-03-18T20:24:40.945147+0000","last_became_peered":"2024-03-18T20:24:40.945147+0000","last_unstale":"2024-03-18T20:24:59.476878+0000","last_undegraded":"2024-03-18T20:24:59.476878+0000","last_fullsized":"2024-03-18T20:24:59.476878+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:45.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436691+0000","last_change":"2024-03-18T20:24:41.366752+0000","last_active":"2024-03-18T20:24:45.436691+0000","last_peered":"2024-03-18T20:24:45.436691+0000","last_clean":"2024-03-18T20:24:45.436691+0000","last_became_active":"2024-03-18T20:24:41.365949+0000","last_became_peered":"2024-03-18T20:24:41.365949+0000","last_unstale":"2024-03-18T20:24:45.436691+0000","last_undegraded":"2024-03-18T20:24:45.436691+0000","last_fullsized":"2024-03-18T20:24:45.436691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:42.785307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962639+0000","last_change":"2024-03-18T20:24:45.446303+0000","last_active":"2024-03-18T20:24:45.962639+0000","last_peered":"2024-03-18T20:24:45.962639+0000","last_clean":"2024-03-18T20:24:45.962639+0000","last_became_active":"2024-03-18T20:24:45.445873+0000","last_became_peered":"2024-03-18T20:24:45.445873+0000","last_unstale":"2024-03-18T20:24:45.962639+0000","last_undegraded":"2024-03-18T20:24:45.962639+0000","last_fullsized":"2024-03-18T20:24:45.962639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:41.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962182+0000","last_change":"2024-03-18T20:24:40.945490+0000","last_active":"2024-03-18T20:24:45.962182+0000","last_peered":"2024-03-18T20:24:45.962182+0000","last_clean":"2024-03-18T20:24:45.962182+0000","last_became_active":"2024-03-18T20:24:40.945155+0000","last_became_peered":"2024-03-18T20:24:40.945155+0000","last_unstale":"2024-03-18T20:24:45.962182+0000","last_undegraded":"2024-03-18T20:24:45.962182+0000","last_fullsized":"2024-03-18T20:24:45.962182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:50.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436214+0000","last_change":"2024-03-18T20:24:41.369354+0000","last_active":"2024-03-18T20:24:45.436214+0000","last_peered":"2024-03-18T20:24:45.436214+0000","last_clean":"2024-03-18T20:24:45.436214+0000","last_became_active":"2024-03-18T20:24:41.369079+0000","last_became_peered":"2024-03-18T20:24:41.369079+0000","last_unstale":"2024-03-18T20:24:45.436214+0000","last_undegraded":"2024-03-18T20:24:45.436214+0000","last_fullsized":"2024-03-18T20:24:45.436214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:25.264037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445168+0000","last_change":"2024-03-18T20:24:45.442906+0000","last_active":"2024-03-18T20:24:45.445168+0000","last_peered":"2024-03-18T20:24:45.445168+0000","last_clean":"2024-03-18T20:24:45.445168+0000","last_became_active":"2024-03-18T20:24:45.442318+0000","last_became_peered":"2024-03-18T20:24:45.442318+0000","last_unstale":"2024-03-18T20:24:45.445168+0000","last_undegraded":"2024-03-18T20:24:45.445168+0000","last_fullsized":"2024-03-18T20:24:45.445168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:50.656133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430336+0000","last_change":"2024-03-18T20:24:40.940614+0000","last_active":"2024-03-18T20:24:45.430336+0000","last_peered":"2024-03-18T20:24:45.430336+0000","last_clean":"2024-03-18T20:24:45.430336+0000","last_became_active":"2024-03-18T20:24:40.940089+0000","last_became_peered":"2024-03-18T20:24:40.940089+0000","last_unstale":"2024-03-18T20:24:45.430336+0000","last_undegraded":"2024-03-18T20:24:45.430336+0000","last_fullsized":"2024-03-18T20:24:45.430336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:09.264556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436344+0000","last_change":"2024-03-18T20:24:41.364100+0000","last_active":"2024-03-18T20:24:45.436344+0000","last_peered":"2024-03-18T20:24:45.436344+0000","last_clean":"2024-03-18T20:24:45.436344+0000","last_became_active":"2024-03-18T20:24:41.363696+0000","last_became_peered":"2024-03-18T20:24:41.363696+0000","last_unstale":"2024-03-18T20:24:45.436344+0000","last_undegraded":"2024-03-18T20:24:45.436344+0000","last_fullsized":"2024-03-18T20:24:45.436344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:46.070917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431227+0000","last_change":"2024-03-18T20:24:41.368853+0000","last_active":"2024-03-18T20:24:45.431227+0000","last_peered":"2024-03-18T20:24:45.431227+0000","last_clean":"2024-03-18T20:24:45.431227+0000","last_became_active":"2024-03-18T20:24:41.368568+0000","last_became_peered":"2024-03-18T20:24:41.368568+0000","last_unstale":"2024-03-18T20:24:45.431227+0000","last_undegraded":"2024-03-18T20:24:45.431227+0000","last_fullsized":"2024-03-18T20:24:45.431227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:24.063361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426949+0000","last_change":"2024-03-18T20:24:41.369055+0000","last_active":"2024-03-18T20:24:45.426949+0000","last_peered":"2024-03-18T20:24:45.426949+0000","last_clean":"2024-03-18T20:24:45.426949+0000","last_became_active":"2024-03-18T20:24:41.368726+0000","last_became_peered":"2024-03-18T20:24:41.368726+0000","last_unstale":"2024-03-18T20:24:45.426949+0000","last_undegraded":"2024-03-18T20:24:45.426949+0000","last_fullsized":"2024-03-18T20:24:45.426949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:15.454756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426328+0000","last_change":"2024-03-18T20:24:41.358811+0000","last_active":"2024-03-18T20:24:45.426328+0000","last_peered":"2024-03-18T20:24:45.426328+0000","last_clean":"2024-03-18T20:24:45.426328+0000","last_became_active":"2024-03-18T20:24:41.357776+0000","last_became_peered":"2024-03-18T20:24:41.357776+0000","last_unstale":"2024-03-18T20:24:45.426328+0000","last_undegraded":"2024-03-18T20:24:45.426328+0000","last_fullsized":"2024-03-18T20:24:45.426328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.069338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442460+0000","last_change":"2024-03-18T20:24:41.370288+0000","last_active":"2024-03-18T20:24:45.442460+0000","last_peered":"2024-03-18T20:24:45.442460+0000","last_clean":"2024-03-18T20:24:45.442460+0000","last_became_active":"2024-03-18T20:24:41.370008+0000","last_became_peered":"2024-03-18T20:24:41.370008+0000","last_unstale":"2024-03-18T20:24:45.442460+0000","last_undegraded":"2024-03-18T20:24:45.442460+0000","last_fullsized":"2024-03-18T20:24:45.442460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:18.246575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430945+0000","last_change":"2024-03-18T20:24:41.370168+0000","last_active":"2024-03-18T20:24:45.430945+0000","last_peered":"2024-03-18T20:24:45.430945+0000","last_clean":"2024-03-18T20:24:45.430945+0000","last_became_active":"2024-03-18T20:24:41.369879+0000","last_became_peered":"2024-03-18T20:24:41.369879+0000","last_unstale":"2024-03-18T20:24:45.430945+0000","last_undegraded":"2024-03-18T20:24:45.430945+0000","last_fullsized":"2024-03-18T20:24:45.430945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:30:42.280869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430414+0000","last_change":"2024-03-18T20:24:41.369036+0000","last_active":"2024-03-18T20:24:45.430414+0000","last_peered":"2024-03-18T20:24:45.430414+0000","last_clean":"2024-03-18T20:24:45.430414+0000","last_became_active":"2024-03-18T20:24:41.368722+0000","last_became_peered":"2024-03-18T20:24:41.368722+0000","last_unstale":"2024-03-18T20:24:45.430414+0000","last_undegraded":"2024-03-18T20:24:45.430414+0000","last_fullsized":"2024-03-18T20:24:45.430414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:11.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427136+0000","last_change":"2024-03-18T20:24:41.355806+0000","last_active":"2024-03-18T20:24:45.427136+0000","last_peered":"2024-03-18T20:24:45.427136+0000","last_clean":"2024-03-18T20:24:45.427136+0000","last_became_active":"2024-03-18T20:24:41.354812+0000","last_became_peered":"2024-03-18T20:24:41.354812+0000","last_unstale":"2024-03-18T20:24:45.427136+0000","last_undegraded":"2024-03-18T20:24:45.427136+0000","last_fullsized":"2024-03-18T20:24:45.427136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:49.307113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426500+0000","last_change":"2024-03-18T20:24:41.371292+0000","last_active":"2024-03-18T20:24:45.426500+0000","last_peered":"2024-03-18T20:24:45.426500+0000","last_clean":"2024-03-18T20:24:45.426500+0000","last_became_active":"2024-03-18T20:24:41.370971+0000","last_became_peered":"2024-03-18T20:24:41.370971+0000","last_unstale":"2024-03-18T20:24:45.426500+0000","last_undegraded":"2024-03-18T20:24:45.426500+0000","last_fullsized":"2024-03-18T20:24:45.426500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:47.568840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438068+0000","last_change":"2024-03-18T20:24:41.368000+0000","last_active":"2024-03-18T20:24:45.438068+0000","last_peered":"2024-03-18T20:24:45.438068+0000","last_clean":"2024-03-18T20:24:45.438068+0000","last_became_active":"2024-03-18T20:24:41.367671+0000","last_became_peered":"2024-03-18T20:24:41.367671+0000","last_unstale":"2024-03-18T20:24:45.438068+0000","last_undegraded":"2024-03-18T20:24:45.438068+0000","last_fullsized":"2024-03-18T20:24:45.438068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:17.940008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425913+0000","last_change":"2024-03-18T20:24:41.354675+0000","last_active":"2024-03-18T20:24:45.425913+0000","last_peered":"2024-03-18T20:24:45.425913+0000","last_clean":"2024-03-18T20:24:45.425913+0000","last_became_active":"2024-03-18T20:24:41.354358+0000","last_became_peered":"2024-03-18T20:24:41.354358+0000","last_unstale":"2024-03-18T20:24:45.425913+0000","last_undegraded":"2024-03-18T20:24:45.425913+0000","last_fullsized":"2024-03-18T20:24:45.425913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:47.238810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433664+0000","last_change":"2024-03-18T20:24:41.360057+0000","last_active":"2024-03-18T20:24:45.433664+0000","last_peered":"2024-03-18T20:24:45.433664+0000","last_clean":"2024-03-18T20:24:45.433664+0000","last_became_active":"2024-03-18T20:24:41.359704+0000","last_became_peered":"2024-03-18T20:24:41.359704+0000","last_unstale":"2024-03-18T20:24:45.433664+0000","last_undegraded":"2024-03-18T20:24:45.433664+0000","last_fullsized":"2024-03-18T20:24:45.433664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:22.235537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430704+0000","last_change":"2024-03-18T20:24:41.357847+0000","last_active":"2024-03-18T20:24:45.430704+0000","last_peered":"2024-03-18T20:24:45.430704+0000","last_clean":"2024-03-18T20:24:45.430704+0000","last_became_active":"2024-03-18T20:24:41.357555+0000","last_became_peered":"2024-03-18T20:24:41.357555+0000","last_unstale":"2024-03-18T20:24:45.430704+0000","last_undegraded":"2024-03-18T20:24:45.430704+0000","last_fullsized":"2024-03-18T20:24:45.430704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:01.049713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431110+0000","last_change":"2024-03-18T20:24:41.366158+0000","last_active":"2024-03-18T20:24:45.431110+0000","last_peered":"2024-03-18T20:24:45.431110+0000","last_clean":"2024-03-18T20:24:45.431110+0000","last_became_active":"2024-03-18T20:24:41.365810+0000","last_became_peered":"2024-03-18T20:24:41.365810+0000","last_unstale":"2024-03-18T20:24:45.431110+0000","last_undegraded":"2024-03-18T20:24:45.431110+0000","last_fullsized":"2024-03-18T20:24:45.431110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:46:39.376215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64936,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":204,"num_read_kb":169,"num_write":154,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514515,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":462381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":9,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":3.5880000000000001,"5min":3.5880000000000001,"15min":3.5880000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"last":0.73899999999999999}]},{"osd":10,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.68200000000000005}]}]},{"osd":10,"up_from":68,"seq":292057776151,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1110742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70999999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.67000000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.623}]},{"osd":6,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.86299999999999999}]},{"osd":9,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.89800000000000002}]},{"osd":11,"last update":"Mon Mar 18 20:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.752},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.78300000000000003}]}]},{"osd":9,"up_from":63,"seq":270582939675,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.433},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.316}]},{"osd":4,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.36899999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.47699999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":8,"up_from":56,"seq":240518168606,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.504,"15min":0.497},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.41799999999999998,"15min":0.40899999999999997},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.441},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46400000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.496},{"interface":"front","average":{"1min":0.621,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.48299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.41299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.68300000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.81000000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.748},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76800000000000002}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.79900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":7,"up_from":49,"seq":210453397538,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1058351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79700000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.496,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.498,"15min":0.5},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.435}]},{"osd":5,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85899999999999999,"5min":0.876,"15min":0.876},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.72999999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.621,"15min":0.621},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.50900000000000001}]},{"osd":8,"last update":"Mon Mar 18 20:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81000000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.79100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69699999999999995}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67300000000000004}]},{"osd":10,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71799999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":6,"up_from":44,"seq":188978561060,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":457793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.372,"15min":0.36899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.27400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.67900000000000005,"15min":0.63300000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":5.508,"5min":5.508,"15min":5.508},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73099999999999998,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.65900000000000003}]},{"osd":8,"last update":"Mon Mar 18 20:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.71499999999999997,"15min":0.68000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":5.5540000000000003,"5min":5.5540000000000003,"15min":5.5540000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.72099999999999997,"15min":0.68500000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":5.5949999999999998,"5min":5.5949999999999998,"15min":5.5949999999999998},"last":0.625}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.75},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57299999999999995}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":516808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.47799999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71399999999999997,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.73299999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.442},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45300000000000001,"15min":0.435},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.436,"15min":0.41699999999999998},"min":{"1min":0.33400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.748},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.878},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.628},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55600000000000005,"15min":0.54600000000000004},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56399999999999995,"15min":0.55700000000000005},"min":{"1min":0.35999999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67200000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.65600000000000003,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90300000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78700000000000003}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65100000000000002}]},{"osd":10,"last update":"Mon Mar 18 20:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":459181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51600000000000001,"15min":0.496},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.71999999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.496},"min":{"1min":0.44,"5min":0.308,"15min":0.308},"max":{"1min":0.65200000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.5,"15min":0.48799999999999999},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53300000000000003,"15min":0.499},"min":{"1min":0.45700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.32800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.37}]},{"osd":4,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.505},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76700000000000002,"5min":1.165,"15min":1.165},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.501,"15min":0.505},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.436}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.60199999999999998}]},{"osd":8,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.42799999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58699999999999997}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.41399999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.49399999999999999}]},{"osd":11,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":518110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.435}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78700000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.312,"15min":0.312},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.498,"15min":0.48899999999999999},"min":{"1min":0.33700000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78200000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.73899999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.39100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64200000000000002}]},{"osd":9,"last update":"Mon Mar 18 20:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.79200000000000004}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.752},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68799999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":457620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33100000000000002},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.255,"5min":0.221,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.437},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.745,"5min":1.256,"15min":1.256},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83399999999999996,"5min":1.228,"15min":1.228},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60099999999999998,"15min":0.59599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.628}]},{"osd":6,"last update":"Mon Mar 18 20:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79700000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.77900000000000003,"15min":0.748},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":7.4390000000000001,"5min":7.4390000000000001,"15min":7.4390000000000001},"last":0.51000000000000001}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.67000000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.376},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.40999999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.452,"5min":7.452,"15min":7.452},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":462787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.49199999999999999,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51000000000000001,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.442}]},{"osd":1,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.33100000000000002},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.377,"15min":0.35299999999999998},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.53500000000000003,"15min":0.49299999999999999},"min":{"1min":0.45500000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.72699999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.39600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71299999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.58399999999999996,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.79000000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.77400000000000002}]},{"osd":8,"last update":"Mon Mar 18 20:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.86099999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.55800000000000005,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96999999999999997,"5min":1.218,"15min":1.218},"last":0.82299999999999995}]},{"osd":9,"last update":"Mon Mar 18 20:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.97099999999999997}]},{"osd":10,"last update":"Mon Mar 18 20:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.89300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1053595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.61399999999999999,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":9.8659999999999997,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":0.5},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.71599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.36199999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69799999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.436},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.307,"5min":0.245,"15min":0.245},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60599999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.59899999999999998,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.2290000000000001},"last":0.76700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70199999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83199999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.879,"15min":0.879},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83199999999999996,"5min":1.254,"15min":1.254},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76200000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.48999999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.441,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77700000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.629},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.498,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47499999999999998}]},{"osd":9,"last update":"Mon Mar 18 20:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.91000000000000003}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:33.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T20:25:33.816 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 -- 172.21.15.98:0/3381247805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20100660 msgr2=0x7fea2010cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 --2- 172.21.15.98:0/3381247805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20100660 0x7fea2010cb00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fea1800b0a0 tx=0x7fea1800b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 -- 172.21.15.98:0/3381247805 shutdown_connections 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 --2- 172.21.15.98:0/3381247805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20100660 0x7fea2010cb00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 --2- 172.21.15.98:0/3381247805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea20105d50 0x7fea20106130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 --2- 172.21.15.98:0/3381247805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea200ffca0 0x7fea20100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 -- 172.21.15.98:0/3381247805 >> 172.21.15.98:0/3381247805 conn(0x7fea20076c80 msgr2=0x7fea20077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 -- 172.21.15.98:0/3381247805 shutdown_connections 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.761+0000 7fea27d30700 1 -- 172.21.15.98:0/3381247805 wait complete. 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 Processor -- start 2024-03-18T20:25:34.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 -- start start 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea200ffca0 0x7fea20197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 0x7fea201984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20198d30 con 0x7fea20105d50 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea201a3540 con 0x7fea200ffca0 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea27d30700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea201a3680 con 0x7fea20100660 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea262cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea262cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34792/0 (socket says 172.21.15.98:34792) 2024-03-18T20:25:34.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea262cd700 1 -- 172.21.15.98:0/3354843326 learned_addr learned my addr 172.21.15.98:0/3354843326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea252cb700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 0x7fea201984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.762+0000 7fea25acc700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea200ffca0 0x7fea20197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 -- 172.21.15.98:0/3354843326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 msgr2=0x7fea201984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 0x7fea201984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 -- 172.21.15.98:0/3354843326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea200ffca0 msgr2=0x7fea20197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea200ffca0 0x7fea20197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 -- 172.21.15.98:0/3354843326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18009d00 con 0x7fea20105d50 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea252cb700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 0x7fea201984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:34.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea262cd700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fea1800b070 tx=0x7fea18009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1800e040 con 0x7fea20105d50 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea201a38b0 con 0x7fea20105d50 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fea1800ce50 con 0x7fea20105d50 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1801b450 con 0x7fea20105d50 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.763+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea201a3db0 con 0x7fea20105d50 2024-03-18T20:25:34.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.764+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea201081e0 con 0x7fea20105d50 2024-03-18T20:25:34.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.765+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fea1801d480 con 0x7fea20105d50 2024-03-18T20:25:34.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.765+0000 7fea12ffd700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 0x7fea0c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.766+0000 7fea25acc700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 0x7fea0c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.766+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fea180ce190 con 0x7fea20105d50 2024-03-18T20:25:34.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.766+0000 7fea25acc700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 0x7fea0c0a3670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea1400abd0 tx=0x7fea1400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:34.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.770+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea18097660 con 0x7fea20105d50 2024-03-18T20:25:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:34 smithi098 ceph-mon[43462]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:34.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:34 smithi098 ceph-mon[43462]: pgmap v249: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:34.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.959+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea20069530 con 0x7fea0c0a11b0 2024-03-18T20:25:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.971+0000 7fea12ffd700 1 -- 172.21.15.98:0/3354843326 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+421945 (secure 0 0 0) 0x7fea20069530 con 0x7fea0c0a11b0 2024-03-18T20:25:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 msgr2=0x7fea0c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 0x7fea0c0a3670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fea1400abd0 tx=0x7fea1400a380 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 msgr2=0x7fea201a3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fea1800b070 tx=0x7fea18009380 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 shutdown_connections 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea20105d50 0x7fea201a3000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fea0c0a11b0 0x7fea0c0a3670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea200ffca0 0x7fea20197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 --2- 172.21.15.98:0/3354843326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea20100660 0x7fea201984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 >> 172.21.15.98:0/3354843326 conn(0x7fea20076c80 msgr2=0x7fea200fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 shutdown_connections 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:34.975+0000 7fea27d30700 1 -- 172.21.15.98:0/3354843326 wait complete. 2024-03-18T20:25:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-18T20:25:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:34 smithi136 ceph-mon[48073]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:34 smithi136 ceph-mon[48073]: pgmap v249: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:34 smithi169 ceph-mon[48529]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:35.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:34 smithi169 ceph-mon[48529]: pgmap v249: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:35.370 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-18T20:25:33.584228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655538,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":340,"num_read_kb":283,"num_write":337,"num_write_kb":3051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337296,"kb_used_data":13104,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604528,"statfs":{"total":1151940427776,"available":1151595036672,"internally_reserved":0,"allocated":13418496,"data_stored":7486287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":37,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430988+0000","last_change":"2024-03-18T20:24:41.360546+0000","last_active":"2024-03-18T20:24:45.430988+0000","last_peered":"2024-03-18T20:24:45.430988+0000","last_clean":"2024-03-18T20:24:45.430988+0000","last_became_active":"2024-03-18T20:24:41.360221+0000","last_became_peered":"2024-03-18T20:24:41.360221+0000","last_unstale":"2024-03-18T20:24:45.430988+0000","last_undegraded":"2024-03-18T20:24:45.430988+0000","last_fullsized":"2024-03-18T20:24:45.430988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:23.329362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441941+0000","last_change":"2024-03-18T20:24:41.354655+0000","last_active":"2024-03-18T20:24:45.441941+0000","last_peered":"2024-03-18T20:24:45.441941+0000","last_clean":"2024-03-18T20:24:45.441941+0000","last_became_active":"2024-03-18T20:24:41.354340+0000","last_became_peered":"2024-03-18T20:24:41.354340+0000","last_unstale":"2024-03-18T20:24:45.441941+0000","last_undegraded":"2024-03-18T20:24:45.441941+0000","last_fullsized":"2024-03-18T20:24:45.441941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:18.785662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440207+0000","last_change":"2024-03-18T20:24:41.351670+0000","last_active":"2024-03-18T20:24:45.440207+0000","last_peered":"2024-03-18T20:24:45.440207+0000","last_clean":"2024-03-18T20:24:45.440207+0000","last_became_active":"2024-03-18T20:24:41.351379+0000","last_became_peered":"2024-03-18T20:24:41.351379+0000","last_unstale":"2024-03-18T20:24:45.440207+0000","last_undegraded":"2024-03-18T20:24:45.440207+0000","last_fullsized":"2024-03-18T20:24:45.440207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:50.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436769+0000","last_change":"2024-03-18T20:24:41.364821+0000","last_active":"2024-03-18T20:24:45.436769+0000","last_peered":"2024-03-18T20:24:45.436769+0000","last_clean":"2024-03-18T20:24:45.436769+0000","last_became_active":"2024-03-18T20:24:41.364496+0000","last_became_peered":"2024-03-18T20:24:41.364496+0000","last_unstale":"2024-03-18T20:24:45.436769+0000","last_undegraded":"2024-03-18T20:24:45.436769+0000","last_fullsized":"2024-03-18T20:24:45.436769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:53.506013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431016+0000","last_change":"2024-03-18T20:24:41.356223+0000","last_active":"2024-03-18T20:24:45.431016+0000","last_peered":"2024-03-18T20:24:45.431016+0000","last_clean":"2024-03-18T20:24:45.431016+0000","last_became_active":"2024-03-18T20:24:41.355831+0000","last_became_peered":"2024-03-18T20:24:41.355831+0000","last_unstale":"2024-03-18T20:24:45.431016+0000","last_undegraded":"2024-03-18T20:24:45.431016+0000","last_fullsized":"2024-03-18T20:24:45.431016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:07.593472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852430+0000","last_change":"2024-03-18T20:24:41.367208+0000","last_active":"2024-03-18T20:24:45.852430+0000","last_peered":"2024-03-18T20:24:45.852430+0000","last_clean":"2024-03-18T20:24:45.852430+0000","last_became_active":"2024-03-18T20:24:41.366885+0000","last_became_peered":"2024-03-18T20:24:41.366885+0000","last_unstale":"2024-03-18T20:24:45.852430+0000","last_undegraded":"2024-03-18T20:24:45.852430+0000","last_fullsized":"2024-03-18T20:24:45.852430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:34.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436116+0000","last_change":"2024-03-18T20:24:41.367935+0000","last_active":"2024-03-18T20:24:45.436116+0000","last_peered":"2024-03-18T20:24:45.436116+0000","last_clean":"2024-03-18T20:24:45.436116+0000","last_became_active":"2024-03-18T20:24:41.367615+0000","last_became_peered":"2024-03-18T20:24:41.367615+0000","last_unstale":"2024-03-18T20:24:45.436116+0000","last_undegraded":"2024-03-18T20:24:45.436116+0000","last_fullsized":"2024-03-18T20:24:45.436116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:41.242857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426070+0000","last_change":"2024-03-18T20:24:41.369045+0000","last_active":"2024-03-18T20:24:45.426070+0000","last_peered":"2024-03-18T20:24:45.426070+0000","last_clean":"2024-03-18T20:24:45.426070+0000","last_became_active":"2024-03-18T20:24:41.368669+0000","last_became_peered":"2024-03-18T20:24:41.368669+0000","last_unstale":"2024-03-18T20:24:45.426070+0000","last_undegraded":"2024-03-18T20:24:45.426070+0000","last_fullsized":"2024-03-18T20:24:45.426070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:09.538243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432055+0000","last_change":"2024-03-18T20:24:41.366520+0000","last_active":"2024-03-18T20:24:45.432055+0000","last_peered":"2024-03-18T20:24:45.432055+0000","last_clean":"2024-03-18T20:24:45.432055+0000","last_became_active":"2024-03-18T20:24:41.366155+0000","last_became_peered":"2024-03-18T20:24:41.366155+0000","last_unstale":"2024-03-18T20:24:45.432055+0000","last_undegraded":"2024-03-18T20:24:45.432055+0000","last_fullsized":"2024-03-18T20:24:45.432055+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:33.151394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852478+0000","last_change":"2024-03-18T20:24:41.370742+0000","last_active":"2024-03-18T20:24:45.852478+0000","last_peered":"2024-03-18T20:24:45.852478+0000","last_clean":"2024-03-18T20:24:45.852478+0000","last_became_active":"2024-03-18T20:24:41.370438+0000","last_became_peered":"2024-03-18T20:24:41.370438+0000","last_unstale":"2024-03-18T20:24:45.852478+0000","last_undegraded":"2024-03-18T20:24:45.852478+0000","last_fullsized":"2024-03-18T20:24:45.852478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:04.331608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436604+0000","last_change":"2024-03-18T20:24:41.366970+0000","last_active":"2024-03-18T20:24:45.436604+0000","last_peered":"2024-03-18T20:24:45.436604+0000","last_clean":"2024-03-18T20:24:45.436604+0000","last_became_active":"2024-03-18T20:24:41.366440+0000","last_became_peered":"2024-03-18T20:24:41.366440+0000","last_unstale":"2024-03-18T20:24:45.436604+0000","last_undegraded":"2024-03-18T20:24:45.436604+0000","last_fullsized":"2024-03-18T20:24:45.436604+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:10.050606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436670+0000","last_change":"2024-03-18T20:24:41.372375+0000","last_active":"2024-03-18T20:24:45.436670+0000","last_peered":"2024-03-18T20:24:45.436670+0000","last_clean":"2024-03-18T20:24:45.436670+0000","last_became_active":"2024-03-18T20:24:41.371028+0000","last_became_peered":"2024-03-18T20:24:41.371028+0000","last_unstale":"2024-03-18T20:24:45.436670+0000","last_undegraded":"2024-03-18T20:24:45.436670+0000","last_fullsized":"2024-03-18T20:24:45.436670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:39.906774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430787+0000","last_change":"2024-03-18T20:24:41.368252+0000","last_active":"2024-03-18T20:24:45.430787+0000","last_peered":"2024-03-18T20:24:45.430787+0000","last_clean":"2024-03-18T20:24:45.430787+0000","last_became_active":"2024-03-18T20:24:41.367911+0000","last_became_peered":"2024-03-18T20:24:41.367911+0000","last_unstale":"2024-03-18T20:24:45.430787+0000","last_undegraded":"2024-03-18T20:24:45.430787+0000","last_fullsized":"2024-03-18T20:24:45.430787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:36.250924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440015+0000","last_change":"2024-03-18T20:24:41.358690+0000","last_active":"2024-03-18T20:24:45.440015+0000","last_peered":"2024-03-18T20:24:45.440015+0000","last_clean":"2024-03-18T20:24:45.440015+0000","last_became_active":"2024-03-18T20:24:41.358254+0000","last_became_peered":"2024-03-18T20:24:41.358254+0000","last_unstale":"2024-03-18T20:24:45.440015+0000","last_undegraded":"2024-03-18T20:24:45.440015+0000","last_fullsized":"2024-03-18T20:24:45.440015+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:51.466799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440570+0000","last_change":"2024-03-18T20:24:41.364337+0000","last_active":"2024-03-18T20:24:45.440570+0000","last_peered":"2024-03-18T20:24:45.440570+0000","last_clean":"2024-03-18T20:24:45.440570+0000","last_became_active":"2024-03-18T20:24:41.364026+0000","last_became_peered":"2024-03-18T20:24:41.364026+0000","last_unstale":"2024-03-18T20:24:45.440570+0000","last_undegraded":"2024-03-18T20:24:45.440570+0000","last_fullsized":"2024-03-18T20:24:45.440570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:25:34.336403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431248+0000","last_change":"2024-03-18T20:24:41.360989+0000","last_active":"2024-03-18T20:24:45.431248+0000","last_peered":"2024-03-18T20:24:45.431248+0000","last_clean":"2024-03-18T20:24:45.431248+0000","last_became_active":"2024-03-18T20:24:41.360097+0000","last_became_peered":"2024-03-18T20:24:41.360097+0000","last_unstale":"2024-03-18T20:24:45.431248+0000","last_undegraded":"2024-03-18T20:24:45.431248+0000","last_fullsized":"2024-03-18T20:24:45.431248+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:18.522013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430879+0000","last_change":"2024-03-18T20:24:41.361221+0000","last_active":"2024-03-18T20:24:45.430879+0000","last_peered":"2024-03-18T20:24:45.430879+0000","last_clean":"2024-03-18T20:24:45.430879+0000","last_became_active":"2024-03-18T20:24:41.360924+0000","last_became_peered":"2024-03-18T20:24:41.360924+0000","last_unstale":"2024-03-18T20:24:45.430879+0000","last_undegraded":"2024-03-18T20:24:45.430879+0000","last_fullsized":"2024-03-18T20:24:45.430879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:24.233761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430265+0000","last_change":"2024-03-18T20:24:41.358145+0000","last_active":"2024-03-18T20:24:45.430265+0000","last_peered":"2024-03-18T20:24:45.430265+0000","last_clean":"2024-03-18T20:24:45.430265+0000","last_became_active":"2024-03-18T20:24:41.357862+0000","last_became_peered":"2024-03-18T20:24:41.357862+0000","last_unstale":"2024-03-18T20:24:45.430265+0000","last_undegraded":"2024-03-18T20:24:45.430265+0000","last_fullsized":"2024-03-18T20:24:45.430265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:01.582221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427416+0000","last_change":"2024-03-18T20:24:41.359214+0000","last_active":"2024-03-18T20:24:45.427416+0000","last_peered":"2024-03-18T20:24:45.427416+0000","last_clean":"2024-03-18T20:24:45.427416+0000","last_became_active":"2024-03-18T20:24:41.357976+0000","last_became_peered":"2024-03-18T20:24:41.357976+0000","last_unstale":"2024-03-18T20:24:45.427416+0000","last_undegraded":"2024-03-18T20:24:45.427416+0000","last_fullsized":"2024-03-18T20:24:45.427416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:36.873383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854870+0000","last_change":"2024-03-18T20:24:45.440427+0000","last_active":"2024-03-18T20:24:45.854870+0000","last_peered":"2024-03-18T20:24:45.854870+0000","last_clean":"2024-03-18T20:24:45.854870+0000","last_became_active":"2024-03-18T20:24:45.440023+0000","last_became_peered":"2024-03-18T20:24:45.440023+0000","last_unstale":"2024-03-18T20:24:45.854870+0000","last_undegraded":"2024-03-18T20:24:45.854870+0000","last_fullsized":"2024-03-18T20:24:45.854870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:37.745854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479395+0000","last_change":"2024-03-18T20:24:40.945829+0000","last_active":"2024-03-18T20:24:59.479395+0000","last_peered":"2024-03-18T20:24:59.479395+0000","last_clean":"2024-03-18T20:24:59.479395+0000","last_became_active":"2024-03-18T20:24:40.945498+0000","last_became_peered":"2024-03-18T20:24:40.945498+0000","last_unstale":"2024-03-18T20:24:59.479395+0000","last_undegraded":"2024-03-18T20:24:59.479395+0000","last_fullsized":"2024-03-18T20:24:59.479395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:40.221678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426446+0000","last_change":"2024-03-18T20:24:41.368911+0000","last_active":"2024-03-18T20:24:45.426446+0000","last_peered":"2024-03-18T20:24:45.426446+0000","last_clean":"2024-03-18T20:24:45.426446+0000","last_became_active":"2024-03-18T20:24:41.368603+0000","last_became_peered":"2024-03-18T20:24:41.368603+0000","last_unstale":"2024-03-18T20:24:45.426446+0000","last_undegraded":"2024-03-18T20:24:45.426446+0000","last_fullsized":"2024-03-18T20:24:45.426446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:37.722935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961438+0000","last_change":"2024-03-18T20:24:45.439904+0000","last_active":"2024-03-18T20:24:45.961438+0000","last_peered":"2024-03-18T20:24:45.961438+0000","last_clean":"2024-03-18T20:24:45.961438+0000","last_became_active":"2024-03-18T20:24:45.439489+0000","last_became_peered":"2024-03-18T20:24:45.439489+0000","last_unstale":"2024-03-18T20:24:45.961438+0000","last_undegraded":"2024-03-18T20:24:45.961438+0000","last_fullsized":"2024-03-18T20:24:45.961438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:21.696148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962086+0000","last_change":"2024-03-18T20:24:40.942150+0000","last_active":"2024-03-18T20:24:45.962086+0000","last_peered":"2024-03-18T20:24:45.962086+0000","last_clean":"2024-03-18T20:24:45.962086+0000","last_became_active":"2024-03-18T20:24:40.941848+0000","last_became_peered":"2024-03-18T20:24:40.941848+0000","last_unstale":"2024-03-18T20:24:45.962086+0000","last_undegraded":"2024-03-18T20:24:45.962086+0000","last_fullsized":"2024-03-18T20:24:45.962086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:44:08.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442127+0000","last_change":"2024-03-18T20:24:41.359112+0000","last_active":"2024-03-18T20:24:45.442127+0000","last_peered":"2024-03-18T20:24:45.442127+0000","last_clean":"2024-03-18T20:24:45.442127+0000","last_became_active":"2024-03-18T20:24:41.358817+0000","last_became_peered":"2024-03-18T20:24:41.358817+0000","last_unstale":"2024-03-18T20:24:45.442127+0000","last_undegraded":"2024-03-18T20:24:45.442127+0000","last_fullsized":"2024-03-18T20:24:45.442127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:25:15.699466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962679+0000","last_change":"2024-03-18T20:24:45.445996+0000","last_active":"2024-03-18T20:24:45.962679+0000","last_peered":"2024-03-18T20:24:45.962679+0000","last_clean":"2024-03-18T20:24:45.962679+0000","last_became_active":"2024-03-18T20:24:45.445552+0000","last_became_peered":"2024-03-18T20:24:45.445552+0000","last_unstale":"2024-03-18T20:24:45.962679+0000","last_undegraded":"2024-03-18T20:24:45.962679+0000","last_fullsized":"2024-03-18T20:24:45.962679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:26.359389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476209+0000","last_change":"2024-03-18T20:24:40.951499+0000","last_active":"2024-03-18T20:24:59.476209+0000","last_peered":"2024-03-18T20:24:59.476209+0000","last_clean":"2024-03-18T20:24:59.476209+0000","last_became_active":"2024-03-18T20:24:40.951183+0000","last_became_peered":"2024-03-18T20:24:40.951183+0000","last_unstale":"2024-03-18T20:24:59.476209+0000","last_undegraded":"2024-03-18T20:24:59.476209+0000","last_fullsized":"2024-03-18T20:24:59.476209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:25.936370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432254+0000","last_change":"2024-03-18T20:24:41.354835+0000","last_active":"2024-03-18T20:24:45.432254+0000","last_peered":"2024-03-18T20:24:45.432254+0000","last_clean":"2024-03-18T20:24:45.432254+0000","last_became_active":"2024-03-18T20:24:41.354486+0000","last_became_peered":"2024-03-18T20:24:41.354486+0000","last_unstale":"2024-03-18T20:24:45.432254+0000","last_undegraded":"2024-03-18T20:24:45.432254+0000","last_fullsized":"2024-03-18T20:24:45.432254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961610+0000","last_change":"2024-03-18T20:24:45.446772+0000","last_active":"2024-03-18T20:24:45.961610+0000","last_peered":"2024-03-18T20:24:45.961610+0000","last_clean":"2024-03-18T20:24:45.961610+0000","last_became_active":"2024-03-18T20:24:45.446296+0000","last_became_peered":"2024-03-18T20:24:45.446296+0000","last_unstale":"2024-03-18T20:24:45.961610+0000","last_undegraded":"2024-03-18T20:24:45.961610+0000","last_fullsized":"2024-03-18T20:24:45.961610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:41:20.784772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431809+0000","last_change":"2024-03-18T20:24:40.948077+0000","last_active":"2024-03-18T20:24:45.431809+0000","last_peered":"2024-03-18T20:24:45.431809+0000","last_clean":"2024-03-18T20:24:45.431809+0000","last_became_active":"2024-03-18T20:24:40.946970+0000","last_became_peered":"2024-03-18T20:24:40.946970+0000","last_unstale":"2024-03-18T20:24:45.431809+0000","last_undegraded":"2024-03-18T20:24:45.431809+0000","last_fullsized":"2024-03-18T20:24:45.431809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:34.558466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437521+0000","last_change":"2024-03-18T20:24:41.371040+0000","last_active":"2024-03-18T20:24:45.437521+0000","last_peered":"2024-03-18T20:24:45.437521+0000","last_clean":"2024-03-18T20:24:45.437521+0000","last_became_active":"2024-03-18T20:24:41.370654+0000","last_became_peered":"2024-03-18T20:24:41.370654+0000","last_unstale":"2024-03-18T20:24:45.437521+0000","last_undegraded":"2024-03-18T20:24:45.437521+0000","last_fullsized":"2024-03-18T20:24:45.437521+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854817+0000","last_change":"2024-03-18T20:24:45.438825+0000","last_active":"2024-03-18T20:24:45.854817+0000","last_peered":"2024-03-18T20:24:45.854817+0000","last_clean":"2024-03-18T20:24:45.854817+0000","last_became_active":"2024-03-18T20:24:45.438418+0000","last_became_peered":"2024-03-18T20:24:45.438418+0000","last_unstale":"2024-03-18T20:24:45.854817+0000","last_undegraded":"2024-03-18T20:24:45.854817+0000","last_fullsized":"2024-03-18T20:24:45.854817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:52.562628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:28.528718+0000","last_change":"2024-03-18T20:24:40.941503+0000","last_active":"2024-03-18T20:25:28.528718+0000","last_peered":"2024-03-18T20:25:28.528718+0000","last_clean":"2024-03-18T20:25:28.528718+0000","last_became_active":"2024-03-18T20:24:40.941150+0000","last_became_peered":"2024-03-18T20:24:40.941150+0000","last_unstale":"2024-03-18T20:25:28.528718+0000","last_undegraded":"2024-03-18T20:25:28.528718+0000","last_fullsized":"2024-03-18T20:25:28.528718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:02.509071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430556+0000","last_change":"2024-03-18T20:24:41.369688+0000","last_active":"2024-03-18T20:24:45.430556+0000","last_peered":"2024-03-18T20:24:45.430556+0000","last_clean":"2024-03-18T20:24:45.430556+0000","last_became_active":"2024-03-18T20:24:41.369361+0000","last_became_peered":"2024-03-18T20:24:41.369361+0000","last_unstale":"2024-03-18T20:24:45.430556+0000","last_undegraded":"2024-03-18T20:24:45.430556+0000","last_fullsized":"2024-03-18T20:24:45.430556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:23.128929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854843+0000","last_change":"2024-03-18T20:24:45.440139+0000","last_active":"2024-03-18T20:24:45.854843+0000","last_peered":"2024-03-18T20:24:45.854843+0000","last_clean":"2024-03-18T20:24:45.854843+0000","last_became_active":"2024-03-18T20:24:45.439758+0000","last_became_peered":"2024-03-18T20:24:45.439758+0000","last_unstale":"2024-03-18T20:24:45.854843+0000","last_undegraded":"2024-03-18T20:24:45.854843+0000","last_fullsized":"2024-03-18T20:24:45.854843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:57.064268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476530+0000","last_change":"2024-03-18T20:24:40.944915+0000","last_active":"2024-03-18T20:24:59.476530+0000","last_peered":"2024-03-18T20:24:59.476530+0000","last_clean":"2024-03-18T20:24:59.476530+0000","last_became_active":"2024-03-18T20:24:40.944547+0000","last_became_peered":"2024-03-18T20:24:40.944547+0000","last_unstale":"2024-03-18T20:24:59.476530+0000","last_undegraded":"2024-03-18T20:24:59.476530+0000","last_fullsized":"2024-03-18T20:24:59.476530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:36.732318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437751+0000","last_change":"2024-03-18T20:24:41.371113+0000","last_active":"2024-03-18T20:24:45.437751+0000","last_peered":"2024-03-18T20:24:45.437751+0000","last_clean":"2024-03-18T20:24:45.437751+0000","last_became_active":"2024-03-18T20:24:41.370764+0000","last_became_peered":"2024-03-18T20:24:41.370764+0000","last_unstale":"2024-03-18T20:24:45.437751+0000","last_undegraded":"2024-03-18T20:24:45.437751+0000","last_fullsized":"2024-03-18T20:24:45.437751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:51.181585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449570+0000","last_change":"2024-03-18T20:24:45.444798+0000","last_active":"2024-03-18T20:24:45.449570+0000","last_peered":"2024-03-18T20:24:45.449570+0000","last_clean":"2024-03-18T20:24:45.449570+0000","last_became_active":"2024-03-18T20:24:45.444374+0000","last_became_peered":"2024-03-18T20:24:45.444374+0000","last_unstale":"2024-03-18T20:24:45.449570+0000","last_undegraded":"2024-03-18T20:24:45.449570+0000","last_fullsized":"2024-03-18T20:24:45.449570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:54.055384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476781+0000","last_change":"2024-03-18T20:24:40.936710+0000","last_active":"2024-03-18T20:24:59.476781+0000","last_peered":"2024-03-18T20:24:59.476781+0000","last_clean":"2024-03-18T20:24:59.476781+0000","last_became_active":"2024-03-18T20:24:40.936360+0000","last_became_peered":"2024-03-18T20:24:40.936360+0000","last_unstale":"2024-03-18T20:24:59.476781+0000","last_undegraded":"2024-03-18T20:24:59.476781+0000","last_fullsized":"2024-03-18T20:24:59.476781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:08.794520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425954+0000","last_change":"2024-03-18T20:24:41.355109+0000","last_active":"2024-03-18T20:24:45.425954+0000","last_peered":"2024-03-18T20:24:45.425954+0000","last_clean":"2024-03-18T20:24:45.425954+0000","last_became_active":"2024-03-18T20:24:41.354718+0000","last_became_peered":"2024-03-18T20:24:41.354718+0000","last_unstale":"2024-03-18T20:24:45.425954+0000","last_undegraded":"2024-03-18T20:24:45.425954+0000","last_fullsized":"2024-03-18T20:24:45.425954+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:39.147267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963393+0000","last_change":"2024-03-18T20:24:45.451275+0000","last_active":"2024-03-18T20:24:45.963393+0000","last_peered":"2024-03-18T20:24:45.963393+0000","last_clean":"2024-03-18T20:24:45.963393+0000","last_became_active":"2024-03-18T20:24:45.450883+0000","last_became_peered":"2024-03-18T20:24:45.450883+0000","last_unstale":"2024-03-18T20:24:45.963393+0000","last_undegraded":"2024-03-18T20:24:45.963393+0000","last_fullsized":"2024-03-18T20:24:45.963393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:10:41.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961539+0000","last_change":"2024-03-18T20:24:40.939692+0000","last_active":"2024-03-18T20:24:45.961539+0000","last_peered":"2024-03-18T20:24:45.961539+0000","last_clean":"2024-03-18T20:24:45.961539+0000","last_became_active":"2024-03-18T20:24:40.939363+0000","last_became_peered":"2024-03-18T20:24:40.939363+0000","last_unstale":"2024-03-18T20:24:45.961539+0000","last_undegraded":"2024-03-18T20:24:45.961539+0000","last_fullsized":"2024-03-18T20:24:45.961539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:41.445337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430758+0000","last_change":"2024-03-18T20:24:41.357835+0000","last_active":"2024-03-18T20:24:45.430758+0000","last_peered":"2024-03-18T20:24:45.430758+0000","last_clean":"2024-03-18T20:24:45.430758+0000","last_became_active":"2024-03-18T20:24:41.357534+0000","last_became_peered":"2024-03-18T20:24:41.357534+0000","last_unstale":"2024-03-18T20:24:45.430758+0000","last_undegraded":"2024-03-18T20:24:45.430758+0000","last_fullsized":"2024-03-18T20:24:45.430758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:16.662257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440398+0000","last_change":"2024-03-18T20:24:45.438295+0000","last_active":"2024-03-18T20:24:45.440398+0000","last_peered":"2024-03-18T20:24:45.440398+0000","last_clean":"2024-03-18T20:24:45.440398+0000","last_became_active":"2024-03-18T20:24:45.437955+0000","last_became_peered":"2024-03-18T20:24:45.437955+0000","last_unstale":"2024-03-18T20:24:45.440398+0000","last_undegraded":"2024-03-18T20:24:45.440398+0000","last_fullsized":"2024-03-18T20:24:45.440398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:26.435144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:29.529509+0000","last_change":"2024-03-18T20:24:40.945507+0000","last_active":"2024-03-18T20:25:29.529509+0000","last_peered":"2024-03-18T20:25:29.529509+0000","last_clean":"2024-03-18T20:25:29.529509+0000","last_became_active":"2024-03-18T20:24:40.945195+0000","last_became_peered":"2024-03-18T20:24:40.945195+0000","last_unstale":"2024-03-18T20:25:29.529509+0000","last_undegraded":"2024-03-18T20:25:29.529509+0000","last_fullsized":"2024-03-18T20:25:29.529509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:29.173130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431923+0000","last_change":"2024-03-18T20:24:41.356795+0000","last_active":"2024-03-18T20:24:45.431923+0000","last_peered":"2024-03-18T20:24:45.431923+0000","last_clean":"2024-03-18T20:24:45.431923+0000","last_became_active":"2024-03-18T20:24:41.356461+0000","last_became_peered":"2024-03-18T20:24:41.356461+0000","last_unstale":"2024-03-18T20:24:45.431923+0000","last_undegraded":"2024-03-18T20:24:45.431923+0000","last_fullsized":"2024-03-18T20:24:45.431923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:04.230985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444215+0000","last_change":"2024-03-18T20:24:45.439984+0000","last_active":"2024-03-18T20:24:45.444215+0000","last_peered":"2024-03-18T20:24:45.444215+0000","last_clean":"2024-03-18T20:24:45.444215+0000","last_became_active":"2024-03-18T20:24:45.439598+0000","last_became_peered":"2024-03-18T20:24:45.439598+0000","last_unstale":"2024-03-18T20:24:45.444215+0000","last_undegraded":"2024-03-18T20:24:45.444215+0000","last_fullsized":"2024-03-18T20:24:45.444215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:49.481812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.469645+0000","last_change":"2024-03-18T20:24:40.943978+0000","last_active":"2024-03-18T20:24:58.469645+0000","last_peered":"2024-03-18T20:24:58.469645+0000","last_clean":"2024-03-18T20:24:58.469645+0000","last_became_active":"2024-03-18T20:24:40.943623+0000","last_became_peered":"2024-03-18T20:24:40.943623+0000","last_unstale":"2024-03-18T20:24:58.469645+0000","last_undegraded":"2024-03-18T20:24:58.469645+0000","last_fullsized":"2024-03-18T20:24:58.469645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:58.995256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442197+0000","last_change":"2024-03-18T20:24:41.365465+0000","last_active":"2024-03-18T20:24:45.442197+0000","last_peered":"2024-03-18T20:24:45.442197+0000","last_clean":"2024-03-18T20:24:45.442197+0000","last_became_active":"2024-03-18T20:24:41.365146+0000","last_became_peered":"2024-03-18T20:24:41.365146+0000","last_unstale":"2024-03-18T20:24:45.442197+0000","last_undegraded":"2024-03-18T20:24:45.442197+0000","last_fullsized":"2024-03-18T20:24:45.442197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:47.903106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962967+0000","last_change":"2024-03-18T20:24:45.440065+0000","last_active":"2024-03-18T20:24:45.962967+0000","last_peered":"2024-03-18T20:24:45.962967+0000","last_clean":"2024-03-18T20:24:45.962967+0000","last_became_active":"2024-03-18T20:24:45.439424+0000","last_became_peered":"2024-03-18T20:24:45.439424+0000","last_unstale":"2024-03-18T20:24:45.962967+0000","last_undegraded":"2024-03-18T20:24:45.962967+0000","last_fullsized":"2024-03-18T20:24:45.962967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:28.350851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471783+0000","last_change":"2024-03-18T20:24:40.942424+0000","last_active":"2024-03-18T20:24:58.471783+0000","last_peered":"2024-03-18T20:24:58.471783+0000","last_clean":"2024-03-18T20:24:58.471783+0000","last_became_active":"2024-03-18T20:24:40.941878+0000","last_became_peered":"2024-03-18T20:24:40.941878+0000","last_unstale":"2024-03-18T20:24:58.471783+0000","last_undegraded":"2024-03-18T20:24:58.471783+0000","last_fullsized":"2024-03-18T20:24:58.471783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:40.746350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.432427+0000","last_change":"2024-03-18T20:24:41.372684+0000","last_active":"2024-03-18T20:24:45.432427+0000","last_peered":"2024-03-18T20:24:45.432427+0000","last_clean":"2024-03-18T20:24:45.432427+0000","last_became_active":"2024-03-18T20:24:41.372171+0000","last_became_peered":"2024-03-18T20:24:41.372171+0000","last_unstale":"2024-03-18T20:24:45.432427+0000","last_undegraded":"2024-03-18T20:24:45.432427+0000","last_fullsized":"2024-03-18T20:24:45.432427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:50.830287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444306+0000","last_change":"2024-03-18T20:24:45.440289+0000","last_active":"2024-03-18T20:24:45.444306+0000","last_peered":"2024-03-18T20:24:45.444306+0000","last_clean":"2024-03-18T20:24:45.444306+0000","last_became_active":"2024-03-18T20:24:45.439910+0000","last_became_peered":"2024-03-18T20:24:45.439910+0000","last_unstale":"2024-03-18T20:24:45.444306+0000","last_undegraded":"2024-03-18T20:24:45.444306+0000","last_fullsized":"2024-03-18T20:24:45.444306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:19.420493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'32","reported_seq":117,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:30.536381+0000","last_change":"2024-03-18T20:24:40.943843+0000","last_active":"2024-03-18T20:25:30.536381+0000","last_peered":"2024-03-18T20:25:30.536381+0000","last_clean":"2024-03-18T20:25:30.536381+0000","last_became_active":"2024-03-18T20:24:40.943521+0000","last_became_peered":"2024-03-18T20:24:40.943521+0000","last_unstale":"2024-03-18T20:25:30.536381+0000","last_undegraded":"2024-03-18T20:25:30.536381+0000","last_fullsized":"2024-03-18T20:25:30.536381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:44.872498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61030,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":69,"num_read_kb":92,"num_write":33,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433854+0000","last_change":"2024-03-18T20:24:41.358139+0000","last_active":"2024-03-18T20:24:45.433854+0000","last_peered":"2024-03-18T20:24:45.433854+0000","last_clean":"2024-03-18T20:24:45.433854+0000","last_became_active":"2024-03-18T20:24:41.357789+0000","last_became_peered":"2024-03-18T20:24:41.357789+0000","last_unstale":"2024-03-18T20:24:45.433854+0000","last_undegraded":"2024-03-18T20:24:45.433854+0000","last_fullsized":"2024-03-18T20:24:45.433854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:49.643210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449427+0000","last_change":"2024-03-18T20:24:45.448899+0000","last_active":"2024-03-18T20:24:45.449427+0000","last_peered":"2024-03-18T20:24:45.449427+0000","last_clean":"2024-03-18T20:24:45.449427+0000","last_became_active":"2024-03-18T20:24:45.448519+0000","last_became_peered":"2024-03-18T20:24:45.448519+0000","last_unstale":"2024-03-18T20:24:45.449427+0000","last_undegraded":"2024-03-18T20:24:45.449427+0000","last_fullsized":"2024-03-18T20:24:45.449427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:54.071495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476499+0000","last_change":"2024-03-18T20:24:40.944455+0000","last_active":"2024-03-18T20:24:59.476499+0000","last_peered":"2024-03-18T20:24:59.476499+0000","last_clean":"2024-03-18T20:24:59.476499+0000","last_became_active":"2024-03-18T20:24:40.944115+0000","last_became_peered":"2024-03-18T20:24:40.944115+0000","last_unstale":"2024-03-18T20:24:59.476499+0000","last_undegraded":"2024-03-18T20:24:59.476499+0000","last_fullsized":"2024-03-18T20:24:59.476499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:06.160375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430066+0000","last_change":"2024-03-18T20:24:41.356074+0000","last_active":"2024-03-18T20:24:45.430066+0000","last_peered":"2024-03-18T20:24:45.430066+0000","last_clean":"2024-03-18T20:24:45.430066+0000","last_became_active":"2024-03-18T20:24:41.355755+0000","last_became_peered":"2024-03-18T20:24:41.355755+0000","last_unstale":"2024-03-18T20:24:45.430066+0000","last_undegraded":"2024-03-18T20:24:45.430066+0000","last_fullsized":"2024-03-18T20:24:45.430066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:54.007791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.443111+0000","last_change":"2024-03-18T20:24:45.440797+0000","last_active":"2024-03-18T20:24:45.443111+0000","last_peered":"2024-03-18T20:24:45.443111+0000","last_clean":"2024-03-18T20:24:45.443111+0000","last_became_active":"2024-03-18T20:24:45.440410+0000","last_became_peered":"2024-03-18T20:24:45.440410+0000","last_unstale":"2024-03-18T20:24:45.443111+0000","last_undegraded":"2024-03-18T20:24:45.443111+0000","last_fullsized":"2024-03-18T20:24:45.443111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:02.586794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.471938+0000","last_change":"2024-03-18T20:24:40.944239+0000","last_active":"2024-03-18T20:24:58.471938+0000","last_peered":"2024-03-18T20:24:58.471938+0000","last_clean":"2024-03-18T20:24:58.471938+0000","last_became_active":"2024-03-18T20:24:40.943845+0000","last_became_peered":"2024-03-18T20:24:40.943845+0000","last_unstale":"2024-03-18T20:24:58.471938+0000","last_undegraded":"2024-03-18T20:24:58.471938+0000","last_fullsized":"2024-03-18T20:24:58.471938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:30.966789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436700+0000","last_change":"2024-03-18T20:24:41.373565+0000","last_active":"2024-03-18T20:24:45.436700+0000","last_peered":"2024-03-18T20:24:45.436700+0000","last_clean":"2024-03-18T20:24:45.436700+0000","last_became_active":"2024-03-18T20:24:41.373201+0000","last_became_peered":"2024-03-18T20:24:41.373201+0000","last_unstale":"2024-03-18T20:24:45.436700+0000","last_undegraded":"2024-03-18T20:24:45.436700+0000","last_fullsized":"2024-03-18T20:24:45.436700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:13.118518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.447664+0000","last_change":"2024-03-18T20:24:45.447115+0000","last_active":"2024-03-18T20:24:45.447664+0000","last_peered":"2024-03-18T20:24:45.447664+0000","last_clean":"2024-03-18T20:24:45.447664+0000","last_became_active":"2024-03-18T20:24:45.446316+0000","last_became_peered":"2024-03-18T20:24:45.446316+0000","last_unstale":"2024-03-18T20:24:45.447664+0000","last_undegraded":"2024-03-18T20:24:45.447664+0000","last_fullsized":"2024-03-18T20:24:45.447664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:17.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:31.532374+0000","last_change":"2024-03-18T20:24:40.938678+0000","last_active":"2024-03-18T20:25:31.532374+0000","last_peered":"2024-03-18T20:25:31.532374+0000","last_clean":"2024-03-18T20:25:31.532374+0000","last_became_active":"2024-03-18T20:24:40.938300+0000","last_became_peered":"2024-03-18T20:24:40.938300+0000","last_unstale":"2024-03-18T20:25:31.532374+0000","last_undegraded":"2024-03-18T20:25:31.532374+0000","last_fullsized":"2024-03-18T20:25:31.532374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:21.806721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.434871+0000","last_change":"2024-03-18T20:24:41.368977+0000","last_active":"2024-03-18T20:24:45.434871+0000","last_peered":"2024-03-18T20:24:45.434871+0000","last_clean":"2024-03-18T20:24:45.434871+0000","last_became_active":"2024-03-18T20:24:41.368545+0000","last_became_peered":"2024-03-18T20:24:41.368545+0000","last_unstale":"2024-03-18T20:24:45.434871+0000","last_undegraded":"2024-03-18T20:24:45.434871+0000","last_fullsized":"2024-03-18T20:24:45.434871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:23.827215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962771+0000","last_change":"2024-03-18T20:24:45.445759+0000","last_active":"2024-03-18T20:24:45.962771+0000","last_peered":"2024-03-18T20:24:45.962771+0000","last_clean":"2024-03-18T20:24:45.962771+0000","last_became_active":"2024-03-18T20:24:45.445366+0000","last_became_peered":"2024-03-18T20:24:45.445366+0000","last_unstale":"2024-03-18T20:24:45.962771+0000","last_undegraded":"2024-03-18T20:24:45.962771+0000","last_fullsized":"2024-03-18T20:24:45.962771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:59.103814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.479110+0000","last_change":"2024-03-18T20:24:40.950980+0000","last_active":"2024-03-18T20:24:59.479110+0000","last_peered":"2024-03-18T20:24:59.479110+0000","last_clean":"2024-03-18T20:24:59.479110+0000","last_became_active":"2024-03-18T20:24:40.950280+0000","last_became_peered":"2024-03-18T20:24:40.950280+0000","last_unstale":"2024-03-18T20:24:59.479110+0000","last_undegraded":"2024-03-18T20:24:59.479110+0000","last_fullsized":"2024-03-18T20:24:59.479110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:16:08.679066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437457+0000","last_change":"2024-03-18T20:24:41.374302+0000","last_active":"2024-03-18T20:24:45.437457+0000","last_peered":"2024-03-18T20:24:45.437457+0000","last_clean":"2024-03-18T20:24:45.437457+0000","last_became_active":"2024-03-18T20:24:41.373974+0000","last_became_peered":"2024-03-18T20:24:41.373974+0000","last_unstale":"2024-03-18T20:24:45.437457+0000","last_undegraded":"2024-03-18T20:24:45.437457+0000","last_fullsized":"2024-03-18T20:24:45.437457+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.251210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445064+0000","last_change":"2024-03-18T20:24:45.443527+0000","last_active":"2024-03-18T20:24:45.445064+0000","last_peered":"2024-03-18T20:24:45.445064+0000","last_clean":"2024-03-18T20:24:45.445064+0000","last_became_active":"2024-03-18T20:24:45.443105+0000","last_became_peered":"2024-03-18T20:24:45.443105+0000","last_unstale":"2024-03-18T20:24:45.445064+0000","last_undegraded":"2024-03-18T20:24:45.445064+0000","last_fullsized":"2024-03-18T20:24:45.445064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:04.568061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.485093+0000","last_change":"2024-03-18T20:24:40.948223+0000","last_active":"2024-03-18T20:25:00.485093+0000","last_peered":"2024-03-18T20:25:00.485093+0000","last_clean":"2024-03-18T20:25:00.485093+0000","last_became_active":"2024-03-18T20:24:40.947889+0000","last_became_peered":"2024-03-18T20:24:40.947889+0000","last_unstale":"2024-03-18T20:25:00.485093+0000","last_undegraded":"2024-03-18T20:25:00.485093+0000","last_fullsized":"2024-03-18T20:25:00.485093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:33.804508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430232+0000","last_change":"2024-03-18T20:24:41.369441+0000","last_active":"2024-03-18T20:24:45.430232+0000","last_peered":"2024-03-18T20:24:45.430232+0000","last_clean":"2024-03-18T20:24:45.430232+0000","last_became_active":"2024-03-18T20:24:41.368271+0000","last_became_peered":"2024-03-18T20:24:41.368271+0000","last_unstale":"2024-03-18T20:24:45.430232+0000","last_undegraded":"2024-03-18T20:24:45.430232+0000","last_fullsized":"2024-03-18T20:24:45.430232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:20.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962427+0000","last_change":"2024-03-18T20:24:45.441117+0000","last_active":"2024-03-18T20:24:45.962427+0000","last_peered":"2024-03-18T20:24:45.962427+0000","last_clean":"2024-03-18T20:24:45.962427+0000","last_became_active":"2024-03-18T20:24:45.440774+0000","last_became_peered":"2024-03-18T20:24:45.440774+0000","last_unstale":"2024-03-18T20:24:45.962427+0000","last_undegraded":"2024-03-18T20:24:45.962427+0000","last_fullsized":"2024-03-18T20:24:45.962427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:03:27.034432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431547+0000","last_change":"2024-03-18T20:24:40.940987+0000","last_active":"2024-03-18T20:24:45.431547+0000","last_peered":"2024-03-18T20:24:45.431547+0000","last_clean":"2024-03-18T20:24:45.431547+0000","last_became_active":"2024-03-18T20:24:40.940652+0000","last_became_peered":"2024-03-18T20:24:40.940652+0000","last_unstale":"2024-03-18T20:24:45.431547+0000","last_undegraded":"2024-03-18T20:24:45.431547+0000","last_fullsized":"2024-03-18T20:24:45.431547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:57.800964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852466+0000","last_change":"2024-03-18T20:24:41.356381+0000","last_active":"2024-03-18T20:24:45.852466+0000","last_peered":"2024-03-18T20:24:45.852466+0000","last_clean":"2024-03-18T20:24:45.852466+0000","last_became_active":"2024-03-18T20:24:41.355078+0000","last_became_peered":"2024-03-18T20:24:41.355078+0000","last_unstale":"2024-03-18T20:24:45.852466+0000","last_undegraded":"2024-03-18T20:24:45.852466+0000","last_fullsized":"2024-03-18T20:24:45.852466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:03.120689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961701+0000","last_change":"2024-03-18T20:24:45.448305+0000","last_active":"2024-03-18T20:24:45.961701+0000","last_peered":"2024-03-18T20:24:45.961701+0000","last_clean":"2024-03-18T20:24:45.961701+0000","last_became_active":"2024-03-18T20:24:45.447706+0000","last_became_peered":"2024-03-18T20:24:45.447706+0000","last_unstale":"2024-03-18T20:24:45.961701+0000","last_undegraded":"2024-03-18T20:24:45.961701+0000","last_fullsized":"2024-03-18T20:24:45.961701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:22.305139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.963464+0000","last_change":"2024-03-18T20:24:40.939482+0000","last_active":"2024-03-18T20:24:45.963464+0000","last_peered":"2024-03-18T20:24:45.963464+0000","last_clean":"2024-03-18T20:24:45.963464+0000","last_became_active":"2024-03-18T20:24:40.938983+0000","last_became_peered":"2024-03-18T20:24:40.938983+0000","last_unstale":"2024-03-18T20:24:45.963464+0000","last_undegraded":"2024-03-18T20:24:45.963464+0000","last_fullsized":"2024-03-18T20:24:45.963464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:00.111633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.852446+0000","last_change":"2024-03-18T20:24:41.366749+0000","last_active":"2024-03-18T20:24:45.852446+0000","last_peered":"2024-03-18T20:24:45.852446+0000","last_clean":"2024-03-18T20:24:45.852446+0000","last_became_active":"2024-03-18T20:24:41.366335+0000","last_became_peered":"2024-03-18T20:24:41.366335+0000","last_unstale":"2024-03-18T20:24:45.852446+0000","last_undegraded":"2024-03-18T20:24:45.852446+0000","last_fullsized":"2024-03-18T20:24:45.852446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:36.927348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962802+0000","last_change":"2024-03-18T20:24:45.447585+0000","last_active":"2024-03-18T20:24:45.962802+0000","last_peered":"2024-03-18T20:24:45.962802+0000","last_clean":"2024-03-18T20:24:45.962802+0000","last_became_active":"2024-03-18T20:24:45.447165+0000","last_became_peered":"2024-03-18T20:24:45.447165+0000","last_unstale":"2024-03-18T20:24:45.962802+0000","last_undegraded":"2024-03-18T20:24:45.962802+0000","last_fullsized":"2024-03-18T20:24:45.962802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:14:34.960675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438282+0000","last_change":"2024-03-18T20:24:41.371969+0000","last_active":"2024-03-18T20:24:45.438282+0000","last_peered":"2024-03-18T20:24:45.438282+0000","last_clean":"2024-03-18T20:24:45.438282+0000","last_became_active":"2024-03-18T20:24:41.371516+0000","last_became_peered":"2024-03-18T20:24:41.371516+0000","last_unstale":"2024-03-18T20:24:45.438282+0000","last_undegraded":"2024-03-18T20:24:45.438282+0000","last_fullsized":"2024-03-18T20:24:45.438282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:46.761356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:30.530337+0000","last_change":"2024-03-18T20:24:40.940375+0000","last_active":"2024-03-18T20:25:30.530337+0000","last_peered":"2024-03-18T20:25:30.530337+0000","last_clean":"2024-03-18T20:25:30.530337+0000","last_became_active":"2024-03-18T20:24:40.940053+0000","last_became_peered":"2024-03-18T20:24:40.940053+0000","last_unstale":"2024-03-18T20:25:30.530337+0000","last_undegraded":"2024-03-18T20:25:30.530337+0000","last_fullsized":"2024-03-18T20:25:30.530337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:20:17.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962803+0000","last_change":"2024-03-18T20:24:45.443315+0000","last_active":"2024-03-18T20:24:45.962803+0000","last_peered":"2024-03-18T20:24:45.962803+0000","last_clean":"2024-03-18T20:24:45.962803+0000","last_became_active":"2024-03-18T20:24:45.442889+0000","last_became_peered":"2024-03-18T20:24:45.442889+0000","last_unstale":"2024-03-18T20:24:45.962803+0000","last_undegraded":"2024-03-18T20:24:45.962803+0000","last_fullsized":"2024-03-18T20:24:45.962803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:40.097397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425997+0000","last_change":"2024-03-18T20:24:41.368952+0000","last_active":"2024-03-18T20:24:45.425997+0000","last_peered":"2024-03-18T20:24:45.425997+0000","last_clean":"2024-03-18T20:24:45.425997+0000","last_became_active":"2024-03-18T20:24:41.368629+0000","last_became_peered":"2024-03-18T20:24:41.368629+0000","last_unstale":"2024-03-18T20:24:45.425997+0000","last_undegraded":"2024-03-18T20:24:45.425997+0000","last_fullsized":"2024-03-18T20:24:45.425997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:28:58.521712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:00.483637+0000","last_change":"2024-03-18T20:24:40.938841+0000","last_active":"2024-03-18T20:25:00.483637+0000","last_peered":"2024-03-18T20:25:00.483637+0000","last_clean":"2024-03-18T20:25:00.483637+0000","last_became_active":"2024-03-18T20:24:40.938464+0000","last_became_peered":"2024-03-18T20:24:40.938464+0000","last_unstale":"2024-03-18T20:25:00.483637+0000","last_undegraded":"2024-03-18T20:25:00.483637+0000","last_fullsized":"2024-03-18T20:25:00.483637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:04.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":326,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962726+0000","last_change":"2024-03-18T20:23:54.597665+0000","last_active":"2024-03-18T20:24:45.962726+0000","last_peered":"2024-03-18T20:24:45.962726+0000","last_clean":"2024-03-18T20:24:45.962726+0000","last_became_active":"2024-03-18T20:23:54.597135+0000","last_became_peered":"2024-03-18T20:23:54.597135+0000","last_unstale":"2024-03-18T20:24:45.962726+0000","last_undegraded":"2024-03-18T20:24:45.962726+0000","last_fullsized":"2024-03-18T20:24:45.962726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:21:34.333460+0000","last_clean_scrub_stamp":"2024-03-18T20:21:34.333460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:59.546138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.449625+0000","last_change":"2024-03-18T20:24:45.448945+0000","last_active":"2024-03-18T20:24:45.449625+0000","last_peered":"2024-03-18T20:24:45.449625+0000","last_clean":"2024-03-18T20:24:45.449625+0000","last_became_active":"2024-03-18T20:24:45.448585+0000","last_became_peered":"2024-03-18T20:24:45.448585+0000","last_unstale":"2024-03-18T20:24:45.449625+0000","last_undegraded":"2024-03-18T20:24:45.449625+0000","last_fullsized":"2024-03-18T20:24:45.449625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:49.668118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.429530+0000","last_change":"2024-03-18T20:24:41.364453+0000","last_active":"2024-03-18T20:24:45.429530+0000","last_peered":"2024-03-18T20:24:45.429530+0000","last_clean":"2024-03-18T20:24:45.429530+0000","last_became_active":"2024-03-18T20:24:41.364047+0000","last_became_peered":"2024-03-18T20:24:41.364047+0000","last_unstale":"2024-03-18T20:24:45.429530+0000","last_undegraded":"2024-03-18T20:24:45.429530+0000","last_fullsized":"2024-03-18T20:24:45.429530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:32.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476629+0000","last_change":"2024-03-18T20:24:40.939926+0000","last_active":"2024-03-18T20:24:59.476629+0000","last_peered":"2024-03-18T20:24:59.476629+0000","last_clean":"2024-03-18T20:24:59.476629+0000","last_became_active":"2024-03-18T20:24:40.939514+0000","last_became_peered":"2024-03-18T20:24:40.939514+0000","last_unstale":"2024-03-18T20:24:59.476629+0000","last_undegraded":"2024-03-18T20:24:59.476629+0000","last_fullsized":"2024-03-18T20:24:59.476629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:41.852567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442992+0000","last_change":"2024-03-18T20:24:45.439880+0000","last_active":"2024-03-18T20:24:45.442992+0000","last_peered":"2024-03-18T20:24:45.442992+0000","last_clean":"2024-03-18T20:24:45.442992+0000","last_became_active":"2024-03-18T20:24:45.439455+0000","last_became_peered":"2024-03-18T20:24:45.439455+0000","last_unstale":"2024-03-18T20:24:45.442992+0000","last_undegraded":"2024-03-18T20:24:45.442992+0000","last_fullsized":"2024-03-18T20:24:45.442992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:26.050401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470311+0000","last_change":"2024-03-18T20:24:40.947764+0000","last_active":"2024-03-18T20:24:58.470311+0000","last_peered":"2024-03-18T20:24:58.470311+0000","last_clean":"2024-03-18T20:24:58.470311+0000","last_became_active":"2024-03-18T20:24:40.946947+0000","last_became_peered":"2024-03-18T20:24:40.946947+0000","last_unstale":"2024-03-18T20:24:58.470311+0000","last_undegraded":"2024-03-18T20:24:58.470311+0000","last_fullsized":"2024-03-18T20:24:58.470311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:38.279134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431036+0000","last_change":"2024-03-18T20:24:41.352213+0000","last_active":"2024-03-18T20:24:45.431036+0000","last_peered":"2024-03-18T20:24:45.431036+0000","last_clean":"2024-03-18T20:24:45.431036+0000","last_became_active":"2024-03-18T20:24:41.351895+0000","last_became_peered":"2024-03-18T20:24:41.351895+0000","last_unstale":"2024-03-18T20:24:45.431036+0000","last_undegraded":"2024-03-18T20:24:45.431036+0000","last_fullsized":"2024-03-18T20:24:45.431036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:55.756782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.440641+0000","last_change":"2024-03-18T20:24:45.439483+0000","last_active":"2024-03-18T20:24:45.440641+0000","last_peered":"2024-03-18T20:24:45.440641+0000","last_clean":"2024-03-18T20:24:45.440641+0000","last_became_active":"2024-03-18T20:24:45.436952+0000","last_became_peered":"2024-03-18T20:24:45.436952+0000","last_unstale":"2024-03-18T20:24:45.440641+0000","last_undegraded":"2024-03-18T20:24:45.440641+0000","last_fullsized":"2024-03-18T20:24:45.440641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:33.839948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:25:05.688972+0000","last_change":"2024-03-18T20:24:40.948184+0000","last_active":"2024-03-18T20:25:05.688972+0000","last_peered":"2024-03-18T20:25:05.688972+0000","last_clean":"2024-03-18T20:25:05.688972+0000","last_became_active":"2024-03-18T20:24:40.947953+0000","last_became_peered":"2024-03-18T20:24:40.947953+0000","last_unstale":"2024-03-18T20:25:05.688972+0000","last_undegraded":"2024-03-18T20:25:05.688972+0000","last_fullsized":"2024-03-18T20:25:05.688972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:51.701653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426390+0000","last_change":"2024-03-18T20:24:41.356563+0000","last_active":"2024-03-18T20:24:45.426390+0000","last_peered":"2024-03-18T20:24:45.426390+0000","last_clean":"2024-03-18T20:24:45.426390+0000","last_became_active":"2024-03-18T20:24:41.356229+0000","last_became_peered":"2024-03-18T20:24:41.356229+0000","last_unstale":"2024-03-18T20:24:45.426390+0000","last_undegraded":"2024-03-18T20:24:45.426390+0000","last_fullsized":"2024-03-18T20:24:45.426390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:20.296131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854824+0000","last_change":"2024-03-18T20:24:45.440937+0000","last_active":"2024-03-18T20:24:45.854824+0000","last_peered":"2024-03-18T20:24:45.854824+0000","last_clean":"2024-03-18T20:24:45.854824+0000","last_became_active":"2024-03-18T20:24:45.440624+0000","last_became_peered":"2024-03-18T20:24:45.440624+0000","last_unstale":"2024-03-18T20:24:45.854824+0000","last_undegraded":"2024-03-18T20:24:45.854824+0000","last_fullsized":"2024-03-18T20:24:45.854824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:41.588153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470417+0000","last_change":"2024-03-18T20:24:40.940622+0000","last_active":"2024-03-18T20:24:58.470417+0000","last_peered":"2024-03-18T20:24:58.470417+0000","last_clean":"2024-03-18T20:24:58.470417+0000","last_became_active":"2024-03-18T20:24:40.940285+0000","last_became_peered":"2024-03-18T20:24:40.940285+0000","last_unstale":"2024-03-18T20:24:58.470417+0000","last_undegraded":"2024-03-18T20:24:58.470417+0000","last_fullsized":"2024-03-18T20:24:58.470417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:03.933412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442035+0000","last_change":"2024-03-18T20:24:41.370212+0000","last_active":"2024-03-18T20:24:45.442035+0000","last_peered":"2024-03-18T20:24:45.442035+0000","last_clean":"2024-03-18T20:24:45.442035+0000","last_became_active":"2024-03-18T20:24:41.369904+0000","last_became_peered":"2024-03-18T20:24:41.369904+0000","last_unstale":"2024-03-18T20:24:45.442035+0000","last_undegraded":"2024-03-18T20:24:45.442035+0000","last_fullsized":"2024-03-18T20:24:45.442035+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:13:12.982929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961796+0000","last_change":"2024-03-18T20:24:45.446935+0000","last_active":"2024-03-18T20:24:45.961796+0000","last_peered":"2024-03-18T20:24:45.961796+0000","last_clean":"2024-03-18T20:24:45.961796+0000","last_became_active":"2024-03-18T20:24:45.445971+0000","last_became_peered":"2024-03-18T20:24:45.445971+0000","last_unstale":"2024-03-18T20:24:45.961796+0000","last_undegraded":"2024-03-18T20:24:45.961796+0000","last_fullsized":"2024-03-18T20:24:45.961796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:51.292743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470479+0000","last_change":"2024-03-18T20:24:40.937625+0000","last_active":"2024-03-18T20:24:58.470479+0000","last_peered":"2024-03-18T20:24:58.470479+0000","last_clean":"2024-03-18T20:24:58.470479+0000","last_became_active":"2024-03-18T20:24:40.937275+0000","last_became_peered":"2024-03-18T20:24:40.937275+0000","last_unstale":"2024-03-18T20:24:58.470479+0000","last_undegraded":"2024-03-18T20:24:58.470479+0000","last_fullsized":"2024-03-18T20:24:58.470479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:55.747307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430187+0000","last_change":"2024-03-18T20:24:41.349528+0000","last_active":"2024-03-18T20:24:45.430187+0000","last_peered":"2024-03-18T20:24:45.430187+0000","last_clean":"2024-03-18T20:24:45.430187+0000","last_became_active":"2024-03-18T20:24:41.349113+0000","last_became_peered":"2024-03-18T20:24:41.349113+0000","last_unstale":"2024-03-18T20:24:45.430187+0000","last_undegraded":"2024-03-18T20:24:45.430187+0000","last_fullsized":"2024-03-18T20:24:45.430187+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:06.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.444468+0000","last_change":"2024-03-18T20:24:45.441827+0000","last_active":"2024-03-18T20:24:45.444468+0000","last_peered":"2024-03-18T20:24:45.444468+0000","last_clean":"2024-03-18T20:24:45.444468+0000","last_became_active":"2024-03-18T20:24:45.441403+0000","last_became_peered":"2024-03-18T20:24:45.441403+0000","last_unstale":"2024-03-18T20:24:45.444468+0000","last_undegraded":"2024-03-18T20:24:45.444468+0000","last_fullsized":"2024-03-18T20:24:45.444468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:37:42.530843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.470136+0000","last_change":"2024-03-18T20:24:40.943702+0000","last_active":"2024-03-18T20:24:58.470136+0000","last_peered":"2024-03-18T20:24:58.470136+0000","last_clean":"2024-03-18T20:24:58.470136+0000","last_became_active":"2024-03-18T20:24:40.943380+0000","last_became_peered":"2024-03-18T20:24:40.943380+0000","last_unstale":"2024-03-18T20:24:58.470136+0000","last_undegraded":"2024-03-18T20:24:58.470136+0000","last_fullsized":"2024-03-18T20:24:58.470136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:04.567157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431139+0000","last_change":"2024-03-18T20:24:41.352181+0000","last_active":"2024-03-18T20:24:45.431139+0000","last_peered":"2024-03-18T20:24:45.431139+0000","last_clean":"2024-03-18T20:24:45.431139+0000","last_became_active":"2024-03-18T20:24:41.351694+0000","last_became_peered":"2024-03-18T20:24:41.351694+0000","last_unstale":"2024-03-18T20:24:45.431139+0000","last_undegraded":"2024-03-18T20:24:45.431139+0000","last_fullsized":"2024-03-18T20:24:45.431139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:00.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:47.625415+0000","last_change":"2024-03-18T20:24:45.444567+0000","last_active":"2024-03-18T20:24:47.625415+0000","last_peered":"2024-03-18T20:24:47.625415+0000","last_clean":"2024-03-18T20:24:47.625415+0000","last_became_active":"2024-03-18T20:24:45.443996+0000","last_became_peered":"2024-03-18T20:24:45.443996+0000","last_unstale":"2024-03-18T20:24:47.625415+0000","last_undegraded":"2024-03-18T20:24:47.625415+0000","last_fullsized":"2024-03-18T20:24:47.625415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:16.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.854563+0000","last_change":"2024-03-18T20:24:40.948872+0000","last_active":"2024-03-18T20:24:45.854563+0000","last_peered":"2024-03-18T20:24:45.854563+0000","last_clean":"2024-03-18T20:24:45.854563+0000","last_became_active":"2024-03-18T20:24:40.948540+0000","last_became_peered":"2024-03-18T20:24:40.948540+0000","last_unstale":"2024-03-18T20:24:45.854563+0000","last_undegraded":"2024-03-18T20:24:45.854563+0000","last_fullsized":"2024-03-18T20:24:45.854563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:57.954340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426231+0000","last_change":"2024-03-18T20:24:41.355552+0000","last_active":"2024-03-18T20:24:45.426231+0000","last_peered":"2024-03-18T20:24:45.426231+0000","last_clean":"2024-03-18T20:24:45.426231+0000","last_became_active":"2024-03-18T20:24:41.355245+0000","last_became_peered":"2024-03-18T20:24:41.355245+0000","last_unstale":"2024-03-18T20:24:45.426231+0000","last_undegraded":"2024-03-18T20:24:45.426231+0000","last_fullsized":"2024-03-18T20:24:45.426231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:25.833815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.961749+0000","last_change":"2024-03-18T20:24:45.443078+0000","last_active":"2024-03-18T20:24:45.961749+0000","last_peered":"2024-03-18T20:24:45.961749+0000","last_clean":"2024-03-18T20:24:45.961749+0000","last_became_active":"2024-03-18T20:24:45.442675+0000","last_became_peered":"2024-03-18T20:24:45.442675+0000","last_unstale":"2024-03-18T20:24:45.961749+0000","last_undegraded":"2024-03-18T20:24:45.961749+0000","last_fullsized":"2024-03-18T20:24:45.961749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:32.262956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:58.472041+0000","last_change":"2024-03-18T20:24:40.943239+0000","last_active":"2024-03-18T20:24:58.472041+0000","last_peered":"2024-03-18T20:24:58.472041+0000","last_clean":"2024-03-18T20:24:58.472041+0000","last_became_active":"2024-03-18T20:24:40.942862+0000","last_became_peered":"2024-03-18T20:24:40.942862+0000","last_unstale":"2024-03-18T20:24:58.472041+0000","last_undegraded":"2024-03-18T20:24:58.472041+0000","last_fullsized":"2024-03-18T20:24:58.472041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:42.560340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.437303+0000","last_change":"2024-03-18T20:24:41.361986+0000","last_active":"2024-03-18T20:24:45.437303+0000","last_peered":"2024-03-18T20:24:45.437303+0000","last_clean":"2024-03-18T20:24:45.437303+0000","last_became_active":"2024-03-18T20:24:41.361608+0000","last_became_peered":"2024-03-18T20:24:41.361608+0000","last_unstale":"2024-03-18T20:24:45.437303+0000","last_undegraded":"2024-03-18T20:24:45.437303+0000","last_fullsized":"2024-03-18T20:24:45.437303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:44.540156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.441527+0000","last_change":"2024-03-18T20:24:45.440250+0000","last_active":"2024-03-18T20:24:45.441527+0000","last_peered":"2024-03-18T20:24:45.441527+0000","last_clean":"2024-03-18T20:24:45.441527+0000","last_became_active":"2024-03-18T20:24:45.439782+0000","last_became_peered":"2024-03-18T20:24:45.439782+0000","last_unstale":"2024-03-18T20:24:45.441527+0000","last_undegraded":"2024-03-18T20:24:45.441527+0000","last_fullsized":"2024-03-18T20:24:45.441527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:58.415354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:59.476878+0000","last_change":"2024-03-18T20:24:40.945951+0000","last_active":"2024-03-18T20:24:59.476878+0000","last_peered":"2024-03-18T20:24:59.476878+0000","last_clean":"2024-03-18T20:24:59.476878+0000","last_became_active":"2024-03-18T20:24:40.945147+0000","last_became_peered":"2024-03-18T20:24:40.945147+0000","last_unstale":"2024-03-18T20:24:59.476878+0000","last_undegraded":"2024-03-18T20:24:59.476878+0000","last_fullsized":"2024-03-18T20:24:59.476878+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:45.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436691+0000","last_change":"2024-03-18T20:24:41.366752+0000","last_active":"2024-03-18T20:24:45.436691+0000","last_peered":"2024-03-18T20:24:45.436691+0000","last_clean":"2024-03-18T20:24:45.436691+0000","last_became_active":"2024-03-18T20:24:41.365949+0000","last_became_peered":"2024-03-18T20:24:41.365949+0000","last_unstale":"2024-03-18T20:24:45.436691+0000","last_undegraded":"2024-03-18T20:24:45.436691+0000","last_fullsized":"2024-03-18T20:24:45.436691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:42.785307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962639+0000","last_change":"2024-03-18T20:24:45.446303+0000","last_active":"2024-03-18T20:24:45.962639+0000","last_peered":"2024-03-18T20:24:45.962639+0000","last_clean":"2024-03-18T20:24:45.962639+0000","last_became_active":"2024-03-18T20:24:45.445873+0000","last_became_peered":"2024-03-18T20:24:45.445873+0000","last_unstale":"2024-03-18T20:24:45.962639+0000","last_undegraded":"2024-03-18T20:24:45.962639+0000","last_fullsized":"2024-03-18T20:24:45.962639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:41.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.962182+0000","last_change":"2024-03-18T20:24:40.945490+0000","last_active":"2024-03-18T20:24:45.962182+0000","last_peered":"2024-03-18T20:24:45.962182+0000","last_clean":"2024-03-18T20:24:45.962182+0000","last_became_active":"2024-03-18T20:24:40.945155+0000","last_became_peered":"2024-03-18T20:24:40.945155+0000","last_unstale":"2024-03-18T20:24:45.962182+0000","last_undegraded":"2024-03-18T20:24:45.962182+0000","last_fullsized":"2024-03-18T20:24:45.962182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:39:50.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436214+0000","last_change":"2024-03-18T20:24:41.369354+0000","last_active":"2024-03-18T20:24:45.436214+0000","last_peered":"2024-03-18T20:24:45.436214+0000","last_clean":"2024-03-18T20:24:45.436214+0000","last_became_active":"2024-03-18T20:24:41.369079+0000","last_became_peered":"2024-03-18T20:24:41.369079+0000","last_unstale":"2024-03-18T20:24:45.436214+0000","last_undegraded":"2024-03-18T20:24:45.436214+0000","last_fullsized":"2024-03-18T20:24:45.436214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:25.264037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.445168+0000","last_change":"2024-03-18T20:24:45.442906+0000","last_active":"2024-03-18T20:24:45.445168+0000","last_peered":"2024-03-18T20:24:45.445168+0000","last_clean":"2024-03-18T20:24:45.445168+0000","last_became_active":"2024-03-18T20:24:45.442318+0000","last_became_peered":"2024-03-18T20:24:45.442318+0000","last_unstale":"2024-03-18T20:24:45.445168+0000","last_undegraded":"2024-03-18T20:24:45.445168+0000","last_fullsized":"2024-03-18T20:24:45.445168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:44.415858+0000","last_clean_scrub_stamp":"2024-03-18T20:24:44.415858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:50.656133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430336+0000","last_change":"2024-03-18T20:24:40.940614+0000","last_active":"2024-03-18T20:24:45.430336+0000","last_peered":"2024-03-18T20:24:45.430336+0000","last_clean":"2024-03-18T20:24:45.430336+0000","last_became_active":"2024-03-18T20:24:40.940089+0000","last_became_peered":"2024-03-18T20:24:40.940089+0000","last_unstale":"2024-03-18T20:24:45.430336+0000","last_undegraded":"2024-03-18T20:24:45.430336+0000","last_fullsized":"2024-03-18T20:24:45.430336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:39.912215+0000","last_clean_scrub_stamp":"2024-03-18T20:24:39.912215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:09.264556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.436344+0000","last_change":"2024-03-18T20:24:41.364100+0000","last_active":"2024-03-18T20:24:45.436344+0000","last_peered":"2024-03-18T20:24:45.436344+0000","last_clean":"2024-03-18T20:24:45.436344+0000","last_became_active":"2024-03-18T20:24:41.363696+0000","last_became_peered":"2024-03-18T20:24:41.363696+0000","last_unstale":"2024-03-18T20:24:45.436344+0000","last_undegraded":"2024-03-18T20:24:45.436344+0000","last_fullsized":"2024-03-18T20:24:45.436344+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:46.070917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431227+0000","last_change":"2024-03-18T20:24:41.368853+0000","last_active":"2024-03-18T20:24:45.431227+0000","last_peered":"2024-03-18T20:24:45.431227+0000","last_clean":"2024-03-18T20:24:45.431227+0000","last_became_active":"2024-03-18T20:24:41.368568+0000","last_became_peered":"2024-03-18T20:24:41.368568+0000","last_unstale":"2024-03-18T20:24:45.431227+0000","last_undegraded":"2024-03-18T20:24:45.431227+0000","last_fullsized":"2024-03-18T20:24:45.431227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:24.063361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426949+0000","last_change":"2024-03-18T20:24:41.369055+0000","last_active":"2024-03-18T20:24:45.426949+0000","last_peered":"2024-03-18T20:24:45.426949+0000","last_clean":"2024-03-18T20:24:45.426949+0000","last_became_active":"2024-03-18T20:24:41.368726+0000","last_became_peered":"2024-03-18T20:24:41.368726+0000","last_unstale":"2024-03-18T20:24:45.426949+0000","last_undegraded":"2024-03-18T20:24:45.426949+0000","last_fullsized":"2024-03-18T20:24:45.426949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:15.454756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426328+0000","last_change":"2024-03-18T20:24:41.358811+0000","last_active":"2024-03-18T20:24:45.426328+0000","last_peered":"2024-03-18T20:24:45.426328+0000","last_clean":"2024-03-18T20:24:45.426328+0000","last_became_active":"2024-03-18T20:24:41.357776+0000","last_became_peered":"2024-03-18T20:24:41.357776+0000","last_unstale":"2024-03-18T20:24:45.426328+0000","last_undegraded":"2024-03-18T20:24:45.426328+0000","last_fullsized":"2024-03-18T20:24:45.426328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.069338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.442460+0000","last_change":"2024-03-18T20:24:41.370288+0000","last_active":"2024-03-18T20:24:45.442460+0000","last_peered":"2024-03-18T20:24:45.442460+0000","last_clean":"2024-03-18T20:24:45.442460+0000","last_became_active":"2024-03-18T20:24:41.370008+0000","last_became_peered":"2024-03-18T20:24:41.370008+0000","last_unstale":"2024-03-18T20:24:45.442460+0000","last_undegraded":"2024-03-18T20:24:45.442460+0000","last_fullsized":"2024-03-18T20:24:45.442460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:18.246575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430945+0000","last_change":"2024-03-18T20:24:41.370168+0000","last_active":"2024-03-18T20:24:45.430945+0000","last_peered":"2024-03-18T20:24:45.430945+0000","last_clean":"2024-03-18T20:24:45.430945+0000","last_became_active":"2024-03-18T20:24:41.369879+0000","last_became_peered":"2024-03-18T20:24:41.369879+0000","last_unstale":"2024-03-18T20:24:45.430945+0000","last_undegraded":"2024-03-18T20:24:45.430945+0000","last_fullsized":"2024-03-18T20:24:45.430945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:30:42.280869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430414+0000","last_change":"2024-03-18T20:24:41.369036+0000","last_active":"2024-03-18T20:24:45.430414+0000","last_peered":"2024-03-18T20:24:45.430414+0000","last_clean":"2024-03-18T20:24:45.430414+0000","last_became_active":"2024-03-18T20:24:41.368722+0000","last_became_peered":"2024-03-18T20:24:41.368722+0000","last_unstale":"2024-03-18T20:24:45.430414+0000","last_undegraded":"2024-03-18T20:24:45.430414+0000","last_fullsized":"2024-03-18T20:24:45.430414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:11.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.427136+0000","last_change":"2024-03-18T20:24:41.355806+0000","last_active":"2024-03-18T20:24:45.427136+0000","last_peered":"2024-03-18T20:24:45.427136+0000","last_clean":"2024-03-18T20:24:45.427136+0000","last_became_active":"2024-03-18T20:24:41.354812+0000","last_became_peered":"2024-03-18T20:24:41.354812+0000","last_unstale":"2024-03-18T20:24:45.427136+0000","last_undegraded":"2024-03-18T20:24:45.427136+0000","last_fullsized":"2024-03-18T20:24:45.427136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:49.307113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.426500+0000","last_change":"2024-03-18T20:24:41.371292+0000","last_active":"2024-03-18T20:24:45.426500+0000","last_peered":"2024-03-18T20:24:45.426500+0000","last_clean":"2024-03-18T20:24:45.426500+0000","last_became_active":"2024-03-18T20:24:41.370971+0000","last_became_peered":"2024-03-18T20:24:41.370971+0000","last_unstale":"2024-03-18T20:24:45.426500+0000","last_undegraded":"2024-03-18T20:24:45.426500+0000","last_fullsized":"2024-03-18T20:24:45.426500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:47.568840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.438068+0000","last_change":"2024-03-18T20:24:41.368000+0000","last_active":"2024-03-18T20:24:45.438068+0000","last_peered":"2024-03-18T20:24:45.438068+0000","last_clean":"2024-03-18T20:24:45.438068+0000","last_became_active":"2024-03-18T20:24:41.367671+0000","last_became_peered":"2024-03-18T20:24:41.367671+0000","last_unstale":"2024-03-18T20:24:45.438068+0000","last_undegraded":"2024-03-18T20:24:45.438068+0000","last_fullsized":"2024-03-18T20:24:45.438068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:17.940008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.425913+0000","last_change":"2024-03-18T20:24:41.354675+0000","last_active":"2024-03-18T20:24:45.425913+0000","last_peered":"2024-03-18T20:24:45.425913+0000","last_clean":"2024-03-18T20:24:45.425913+0000","last_became_active":"2024-03-18T20:24:41.354358+0000","last_became_peered":"2024-03-18T20:24:41.354358+0000","last_unstale":"2024-03-18T20:24:45.425913+0000","last_undegraded":"2024-03-18T20:24:45.425913+0000","last_fullsized":"2024-03-18T20:24:45.425913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:47.238810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.433664+0000","last_change":"2024-03-18T20:24:41.360057+0000","last_active":"2024-03-18T20:24:45.433664+0000","last_peered":"2024-03-18T20:24:45.433664+0000","last_clean":"2024-03-18T20:24:45.433664+0000","last_became_active":"2024-03-18T20:24:41.359704+0000","last_became_peered":"2024-03-18T20:24:41.359704+0000","last_unstale":"2024-03-18T20:24:45.433664+0000","last_undegraded":"2024-03-18T20:24:45.433664+0000","last_fullsized":"2024-03-18T20:24:45.433664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:22.235537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.430704+0000","last_change":"2024-03-18T20:24:41.357847+0000","last_active":"2024-03-18T20:24:45.430704+0000","last_peered":"2024-03-18T20:24:45.430704+0000","last_clean":"2024-03-18T20:24:45.430704+0000","last_became_active":"2024-03-18T20:24:41.357555+0000","last_became_peered":"2024-03-18T20:24:41.357555+0000","last_unstale":"2024-03-18T20:24:45.430704+0000","last_undegraded":"2024-03-18T20:24:45.430704+0000","last_fullsized":"2024-03-18T20:24:45.430704+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:01.049713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-18T20:24:45.431110+0000","last_change":"2024-03-18T20:24:41.366158+0000","last_active":"2024-03-18T20:24:45.431110+0000","last_peered":"2024-03-18T20:24:45.431110+0000","last_clean":"2024-03-18T20:24:45.431110+0000","last_became_active":"2024-03-18T20:24:41.365810+0000","last_became_peered":"2024-03-18T20:24:41.365810+0000","last_unstale":"2024-03-18T20:24:45.431110+0000","last_undegraded":"2024-03-18T20:24:45.431110+0000","last_fullsized":"2024-03-18T20:24:45.431110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:24:40.921134+0000","last_clean_scrub_stamp":"2024-03-18T20:24:40.921134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:46:39.376215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64936,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":214,"num_read_kb":174,"num_write":154,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514515,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":462381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":9,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":3.5880000000000001,"5min":3.5880000000000001,"15min":3.5880000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"last":0.73899999999999999}]},{"osd":10,"last update":"Mon Mar 18 20:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.68200000000000005}]}]},{"osd":10,"up_from":68,"seq":292057776151,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1110742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70999999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.67000000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.623}]},{"osd":6,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":8,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.86299999999999999}]},{"osd":9,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.89800000000000002}]},{"osd":11,"last update":"Mon Mar 18 20:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.752},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.78300000000000003}]}]},{"osd":9,"up_from":63,"seq":270582939675,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.433},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.316}]},{"osd":4,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.36899999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.47699999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":8,"up_from":56,"seq":240518168606,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.504,"15min":0.497},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.41799999999999998,"15min":0.40899999999999997},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.441},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46400000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72399999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.496},{"interface":"front","average":{"1min":0.621,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.48299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.41299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.68300000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.81000000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.748},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76800000000000002}]},{"osd":10,"last update":"Mon Mar 18 20:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.79900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":7,"up_from":49,"seq":210453397538,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1058351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79700000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.496,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.498,"15min":0.5},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.435}]},{"osd":5,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85899999999999999,"5min":0.876,"15min":0.876},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.72999999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.621,"15min":0.621},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.50900000000000001}]},{"osd":8,"last update":"Mon Mar 18 20:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81000000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.79100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69699999999999995}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67300000000000004}]},{"osd":10,"last update":"Mon Mar 18 20:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71799999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":6,"up_from":44,"seq":188978561061,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":457793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.372,"15min":0.36899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.317},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.27400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.38},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.46000000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44400000000000001,"15min":0.442},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.495},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.67900000000000005,"15min":0.63300000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":5.508,"5min":5.508,"15min":5.508},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.621},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.60299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73099999999999998,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.51600000000000001}]},{"osd":8,"last update":"Mon Mar 18 20:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.71499999999999997,"15min":0.68000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":5.5540000000000003,"5min":5.5540000000000003,"15min":5.5540000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.72099999999999997,"15min":0.68500000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":5.5949999999999998,"5min":5.5949999999999998,"15min":5.5949999999999998},"last":0.58099999999999996}]},{"osd":9,"last update":"Mon Mar 18 20:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.502}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":516808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.47799999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71399999999999997,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.73299999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.442},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45300000000000001,"15min":0.435},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.436,"15min":0.41699999999999998},"min":{"1min":0.33400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58599999999999997,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.748},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.878},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.628},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55600000000000005,"15min":0.54600000000000004},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56399999999999995,"15min":0.55700000000000005},"min":{"1min":0.35999999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67200000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.65600000000000003,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90300000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78700000000000003}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65100000000000002}]},{"osd":10,"last update":"Mon Mar 18 20:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":459181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51600000000000001,"15min":0.496},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.71999999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.496},"min":{"1min":0.44,"5min":0.308,"15min":0.308},"max":{"1min":0.65200000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.5,"15min":0.48799999999999999},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53300000000000003,"15min":0.499},"min":{"1min":0.45700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59499999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.32800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.437},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.69699999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:30 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85599999999999998,"5min":0.872,"15min":0.872},"last":0.623}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76700000000000002,"5min":1.165,"15min":1.165},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.501,"15min":0.505},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39500000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76700000000000002}]},{"osd":8,"last update":"Mon Mar 18 20:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.42799999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.68899999999999995,"15min":0.69499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.68500000000000005}]},{"osd":9,"last update":"Mon Mar 18 20:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.89400000000000002}]},{"osd":10,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.61099999999999999}]},{"osd":11,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66000000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":518110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.435}]},{"osd":1,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78700000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.312,"15min":0.312},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.498,"15min":0.48899999999999999},"min":{"1min":0.33700000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78200000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.73899999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.39100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64200000000000002}]},{"osd":9,"last update":"Mon Mar 18 20:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.79200000000000004}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.752},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68799999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":457620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33100000000000002},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.255,"5min":0.221,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.437},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.745,"5min":1.256,"15min":1.256},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83399999999999996,"5min":1.228,"15min":1.228},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.61899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60099999999999998,"15min":0.59599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.56999999999999995,"15min":0.54400000000000004},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.628}]},{"osd":6,"last update":"Mon Mar 18 20:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79700000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.77900000000000003,"15min":0.748},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":7.4390000000000001,"5min":7.4390000000000001,"15min":7.4390000000000001},"last":0.51000000000000001}]},{"osd":8,"last update":"Mon Mar 18 20:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.67000000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.376},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.40999999999999998}]},{"osd":10,"last update":"Mon Mar 18 20:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.452,"5min":7.452,"15min":7.452},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":462787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.49199999999999999,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51000000000000001,"15min":0.495},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.442}]},{"osd":1,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.33100000000000002},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.377,"15min":0.35299999999999998},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.53500000000000003,"15min":0.49299999999999999},"min":{"1min":0.45500000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.68300000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.39600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.73499999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.58399999999999996,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.58899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":7,"last update":"Mon Mar 18 20:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.61699999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.86099999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.55800000000000005,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96999999999999997,"5min":1.218,"15min":1.218},"last":0.67100000000000004}]},{"osd":9,"last update":"Mon Mar 18 20:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.747}]},{"osd":10,"last update":"Mon Mar 18 20:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.76000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1053595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.61399999999999999,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":9.8659999999999997,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":0.5},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.71599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.36199999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69799999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.436},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.378},"min":{"1min":0.307,"5min":0.245,"15min":0.245},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60599999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.59899999999999998,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.2290000000000001},"last":0.76700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70199999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83199999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.73299999999999998,"5min":0.879,"15min":0.879},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83199999999999996,"5min":1.254,"15min":1.254},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76200000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.48999999999999999}]},{"osd":8,"last update":"Mon Mar 18 20:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.441,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77700000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.629},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.498,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47499999999999998}]},{"osd":9,"last update":"Mon Mar 18 20:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.91000000000000003}]},{"osd":10,"last update":"Mon Mar 18 20:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:35.378 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T20:25:35.378 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T20:25:35.378 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T20:25:35.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T20:25:35.603 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 -- 172.21.15.98:0/52786683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0699d0 msgr2=0x7f049c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/52786683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0699d0 0x7f049c10dc10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f049400b0d0 tx=0x7f049400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 -- 172.21.15.98:0/52786683 shutdown_connections 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/52786683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c0699d0 0x7f049c10dc10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/52786683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c106df0 0x7f049c1071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/52786683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 0x7f049c069490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 -- 172.21.15.98:0/52786683 >> 172.21.15.98:0/52786683 conn(0x7f049c076c80 msgr2=0x7f049c077090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 -- 172.21.15.98:0/52786683 shutdown_connections 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.602+0000 7f04a2ebc700 1 -- 172.21.15.98:0/52786683 wait complete. 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 Processor -- start 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 -- start start 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 0x7f049c1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c0699d0 0x7f049c197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19c0f0 con 0x7f049c106df0 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19c260 con 0x7f049c069010 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a2ebc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19c420 con 0x7f049c0699d0 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34816/0 (socket says 172.21.15.98:34816) 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 -- 172.21.15.98:0/3280897426 learned_addr learned my addr 172.21.15.98:0/3280897426 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a0c58700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 0x7f049c1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f049bfff700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c0699d0 0x7f049c197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 -- 172.21.15.98:0/3280897426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c0699d0 msgr2=0x7f049c197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c0699d0 0x7f049c197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 -- 172.21.15.98:0/3280897426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 msgr2=0x7f049c1971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 0x7f049c1971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.603+0000 7f04a1459700 1 -- 172.21.15.98:0/3280897426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494009d00 con 0x7f049c106df0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f04a1459700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0494007930 tx=0x7f0494007960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400e040 con 0x7f049c106df0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c19c6a0 con 0x7f049c106df0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0494003e90 con 0x7f049c106df0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494005930 con 0x7f049c106df0 2024-03-18T20:25:36.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.604+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c1a3f60 con 0x7f049c106df0 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.605+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c1092f0 con 0x7f049c106df0 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.606+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f0494003a00 con 0x7f049c106df0 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.606+0000 7f0499ffb700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 0x7f048c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.606+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f04940cdda0 con 0x7f049c106df0 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.606+0000 7f04a0c58700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 0x7f048c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.607+0000 7f04a0c58700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 0x7f048c0a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0490005950 tx=0x7f049000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.611+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0494097270 con 0x7f049c106df0 2024-03-18T20:25:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.849+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f049c04ea80 con 0x7f049c106df0 2024-03-18T20:25:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.849+0000 7f0499ffb700 1 -- 172.21.15.98:0/3280897426 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f049409a890 con 0x7f049c106df0 2024-03-18T20:25:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:36.854 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 msgr2=0x7f048c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 0x7f048c0a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0490005950 tx=0x7f049000b410 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 msgr2=0x7f049c19bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0494007930 tx=0x7f0494007960 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 shutdown_connections 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f049c106df0 0x7f049c19bae0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f048c0a11b0 0x7f048c0a3670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f049c069010 0x7f049c1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 --2- 172.21.15.98:0/3280897426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f049c0699d0 0x7f049c197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 >> 172.21.15.98:0/3280897426 conn(0x7f049c076c80 msgr2=0x7f049c0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:36.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 shutdown_connections 2024-03-18T20:25:36.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:36.851+0000 7f04a2ebc700 1 -- 172.21.15.98:0/3280897426 wait complete. 2024-03-18T20:25:36.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:36 smithi136 ceph-mon[48073]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:36.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:36 smithi136 ceph-mon[48073]: pgmap v250: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:36.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:36 smithi098 ceph-mon[43462]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:36.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:36 smithi098 ceph-mon[43462]: pgmap v250: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:36 smithi169 ceph-mon[48529]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T20:25:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:36 smithi169 ceph-mon[48529]: pgmap v250: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:37.220 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T20:25:37.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 -- 172.21.15.98:0/531648702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a41016e0 msgr2=0x7f93a4103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 --2- 172.21.15.98:0/531648702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a41016e0 0x7f93a4103ad0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9394009990 tx=0x7f9394009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 -- 172.21.15.98:0/531648702 shutdown_connections 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 --2- 172.21.15.98:0/531648702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a4106940 0x7f93a4108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 --2- 172.21.15.98:0/531648702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a41016e0 0x7f93a4103ad0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 --2- 172.21.15.98:0/531648702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 0x7f93a4106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 -- 172.21.15.98:0/531648702 >> 172.21.15.98:0/531648702 conn(0x7f93a40fb0d0 msgr2=0x7f93a40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 -- 172.21.15.98:0/531648702 shutdown_connections 2024-03-18T20:25:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.342+0000 7f93a9d86700 1 -- 172.21.15.98:0/531648702 wait complete. 2024-03-18T20:25:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 Processor -- start 2024-03-18T20:25:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 -- start start 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a41016e0 0x7f93a4196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 0x7f93a41970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a419e6d0 con 0x7f93a4106940 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a419e810 con 0x7f93a4104010 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a9d86700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a419e9a0 con 0x7f93a41016e0 2024-03-18T20:25:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34826/0 (socket says 172.21.15.98:34826) 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a3fff700 1 -- 172.21.15.98:0/554202317 learned_addr learned my addr 172.21.15.98:0/554202317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a37fe700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a41016e0 0x7f93a4196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.343+0000 7f93a2ffd700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 0x7f93a41970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 -- 172.21.15.98:0/554202317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a41016e0 msgr2=0x7f93a4196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a41016e0 0x7f93a4196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 -- 172.21.15.98:0/554202317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 msgr2=0x7f93a41970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 0x7f93a41970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 -- 172.21.15.98:0/554202317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940095f0 con 0x7f93a4106940 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a3fff700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f939c007ee0 tx=0x7f939c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c017070 con 0x7f93a4106940 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a419ec80 con 0x7f93a4106940 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f939c00f040 con 0x7f93a4106940 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c005570 con 0x7f93a4106940 2024-03-18T20:25:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.344+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a419f190 con 0x7f93a4106940 2024-03-18T20:25:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.345+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40fcc50 con 0x7f93a4106940 2024-03-18T20:25:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.347+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f939c00db20 con 0x7f93a4106940 2024-03-18T20:25:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.347+0000 7f93a0ff9700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 0x7f938c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.347+0000 7f93a37fe700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 0x7f938c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.347+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f939c0cc9d0 con 0x7f93a4106940 2024-03-18T20:25:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.348+0000 7f93a37fe700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 0x7f938c0a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9394003780 tx=0x7f9394005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.351+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f939c095e10 con 0x7f93a4106940 2024-03-18T20:25:37.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.562+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93a402ccb0 con 0x7f93a4106940 2024-03-18T20:25:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.564+0000 7f93a0ff9700 1 -- 172.21.15.98:0/554202317 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f939c099430 con 0x7f93a4106940 2024-03-18T20:25:37.569 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:37.569 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 msgr2=0x7f938c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 0x7f938c0a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9394003780 tx=0x7f9394005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 msgr2=0x7f93a419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f939c007ee0 tx=0x7f939c00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 shutdown_connections 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4106940 0x7f93a419e100 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f93a41016e0 0x7f93a4196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f938c0a11b0 0x7f938c0a3670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 --2- 172.21.15.98:0/554202317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93a4104010 0x7f93a41970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 >> 172.21.15.98:0/554202317 conn(0x7f93a40fb0d0 msgr2=0x7f93a41036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 shutdown_connections 2024-03-18T20:25:37.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.566+0000 7f93a9d86700 1 -- 172.21.15.98:0/554202317 wait complete. 2024-03-18T20:25:37.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:25:37.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:37.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 -- 172.21.15.98:0/3398070756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 msgr2=0x7fbb001080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 --2- 172.21.15.98:0/3398070756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001080d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800b0d0 tx=0x7fbaf800b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 -- 172.21.15.98:0/3398070756 shutdown_connections 2024-03-18T20:25:37.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 --2- 172.21.15.98:0/3398070756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001080d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 --2- 172.21.15.98:0/3398070756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 0x7fbb001000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 --2- 172.21.15.98:0/3398070756 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb001012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 -- 172.21.15.98:0/3398070756 >> 172.21.15.98:0/3398070756 conn(0x7fbb000fb200 msgr2=0x7fbb000fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 -- 172.21.15.98:0/3398070756 shutdown_connections 2024-03-18T20:25:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.709+0000 7fbb057de700 1 -- 172.21.15.98:0/3398070756 wait complete. 2024-03-18T20:25:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 Processor -- start 2024-03-18T20:25:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 -- start start 2024-03-18T20:25:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 0x7fbb00103b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb00102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb00102e50 con 0x7fbb00101820 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb00102fc0 con 0x7fbb000ffc90 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbb057de700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001a0e60 con 0x7fbb00100e80 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbaff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbaff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34840/0 (socket says 172.21.15.98:34840) 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbaff7fe700 1 -- 172.21.15.98:0/251774406 learned_addr learned my addr 172.21.15.98:0/251774406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbafeffd700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 0x7fbb00103b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.710+0000 7fbafe7fc700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb00102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 -- 172.21.15.98:0/251774406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 msgr2=0x7fbb00102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb00102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 -- 172.21.15.98:0/251774406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 msgr2=0x7fbb00103b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 0x7fbb00103b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 -- 172.21.15.98:0/251774406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf8009d00 con 0x7fbb00101820 2024-03-18T20:25:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbafe7fc700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb00102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:37.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbaff7fe700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8007b60 tx=0x7fbaf80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf800e040 con 0x7fbb00101820 2024-03-18T20:25:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbaf8012d90 con 0x7fbb00101820 2024-03-18T20:25:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb001a1000 con 0x7fbb00101820 2024-03-18T20:25:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf801de60 con 0x7fbb00101820 2024-03-18T20:25:37.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.711+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb001a1530 con 0x7fbb00101820 2024-03-18T20:25:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.712+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00067f40 con 0x7fbb00101820 2024-03-18T20:25:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.713+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbaf8012900 con 0x7fbb00101820 2024-03-18T20:25:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.714+0000 7fbae7fff700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 0x7fbae80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.714+0000 7fbafeffd700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 0x7fbae80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.714+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbaf80ced50 con 0x7fbb00101820 2024-03-18T20:25:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.714+0000 7fbafeffd700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 0x7fbae80a36f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbb00100ce0 tx=0x7fbaf400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.717+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbaf8098220 con 0x7fbb00101820 2024-03-18T20:25:37.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:37 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3280897426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T20:25:37.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:37 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/554202317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.915+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb001a1750 con 0x7fbb00101820 2024-03-18T20:25:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.915+0000 7fbae7fff700 1 -- 172.21.15.98:0/251774406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbaf8021020 con 0x7fbb00101820 2024-03-18T20:25:37.920 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 msgr2=0x7fbae80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 0x7fbae80a36f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbb00100ce0 tx=0x7fbaf400b410 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 msgr2=0x7fbb001027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8007b60 tx=0x7fbaf80094c0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 shutdown_connections 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb00101820 0x7fbb001027a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbae80a1230 0x7fbae80a36f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb000ffc90 0x7fbb00103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 --2- 172.21.15.98:0/251774406 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb00100e80 0x7fbb00102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 >> 172.21.15.98:0/251774406 conn(0x7fbb000fb200 msgr2=0x7fbb00069380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 shutdown_connections 2024-03-18T20:25:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:37.919+0000 7fbb057de700 1 -- 172.21.15.98:0/251774406 wait complete. 2024-03-18T20:25:37.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:37 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3280897426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T20:25:37.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:37 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/554202317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:38.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 -- 172.21.15.98:0/2891390901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274100e80 msgr2=0x7fc2741012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 --2- 172.21.15.98:0/2891390901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274100e80 0x7fc2741012e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc2600099c0 tx=0x7fc260009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 -- 172.21.15.98:0/2891390901 shutdown_connections 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 --2- 172.21.15.98:0/2891390901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc274101820 0x7fc2741080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 --2- 172.21.15.98:0/2891390901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2740ffc90 0x7fc2741000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 --2- 172.21.15.98:0/2891390901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274100e80 0x7fc2741012e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 -- 172.21.15.98:0/2891390901 >> 172.21.15.98:0/2891390901 conn(0x7fc2740fb200 msgr2=0x7fc2740fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 -- 172.21.15.98:0/2891390901 shutdown_connections 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.056+0000 7fc27b251700 1 -- 172.21.15.98:0/2891390901 wait complete. 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 Processor -- start 2024-03-18T20:25:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 -- start start 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 0x7fc274195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc274100e80 0x7fc274196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27419d660 con 0x7fc274101820 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27419d7a0 con 0x7fc274100e80 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.057+0000 7fc27b251700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27419d930 con 0x7fc2740ffc90 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34856/0 (socket says 172.21.15.98:34856) 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 -- 172.21.15.98:0/3786911127 learned_addr learned my addr 172.21.15.98:0/3786911127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc273fff700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc274100e80 0x7fc274196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc278fed700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 0x7fc274195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 -- 172.21.15.98:0/3786911127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 msgr2=0x7fc274195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 0x7fc274195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 -- 172.21.15.98:0/3786911127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc274100e80 msgr2=0x7fc274196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc274100e80 0x7fc274196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 -- 172.21.15.98:0/3786911127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600095f0 con 0x7fc274101820 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc278fed700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 0x7fc274195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc2797ee700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00ef30 tx=0x7fc26c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c018070 con 0x7fc274101820 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc26c00f040 con 0x7fc274101820 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c00cc70 con 0x7fc274101820 2024-03-18T20:25:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27419dbb0 con 0x7fc274101820 2024-03-18T20:25:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.058+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27419e070 con 0x7fc274101820 2024-03-18T20:25:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.059+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2741071d0 con 0x7fc274101820 2024-03-18T20:25:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.061+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc26c003a00 con 0x7fc274101820 2024-03-18T20:25:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.062+0000 7fc271ffb700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 0x7fc2640a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.062+0000 7fc278fed700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 0x7fc2640a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.062+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc26c0cce00 con 0x7fc274101820 2024-03-18T20:25:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.062+0000 7fc278fed700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 0x7fc2640a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc274100ce0 tx=0x7fc268009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.066+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc26c096270 con 0x7fc274101820 2024-03-18T20:25:38.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:37 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3280897426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T20:25:38.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:37 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/554202317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.260+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc27402cdc0 con 0x7fc274101820 2024-03-18T20:25:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.260+0000 7fc271ffb700 1 -- 172.21.15.98:0/3786911127 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc26c003cb0 con 0x7fc274101820 2024-03-18T20:25:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 msgr2=0x7fc2640a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 0x7fc2640a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc274100ce0 tx=0x7fc268009260 comp rx=0 tx=0).stop 2024-03-18T20:25:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 msgr2=0x7fc27419d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00ef30 tx=0x7fc26c00c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 shutdown_connections 2024-03-18T20:25:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc274101820 0x7fc27419d090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2740ffc90 0x7fc274195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc2640a11b0 0x7fc2640a3670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 --2- 172.21.15.98:0/3786911127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc274100e80 0x7fc274196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 >> 172.21.15.98:0/3786911127 conn(0x7fc2740fb200 msgr2=0x7fc274104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 shutdown_connections 2024-03-18T20:25:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.263+0000 7fc27b251700 1 -- 172.21.15.98:0/3786911127 wait complete. 2024-03-18T20:25:38.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 -- 172.21.15.98:0/2182796287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e740ffec0 msgr2=0x7f9e74100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/2182796287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e740ffec0 0x7f9e74100340 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009990 tx=0x7f9e64009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 -- 172.21.15.98:0/2182796287 shutdown_connections 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/2182796287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74100880 0x7f9e74108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/2182796287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e74101270 0x7f9e74101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/2182796287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e740ffec0 0x7f9e74100340 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 -- 172.21.15.98:0/2182796287 >> 172.21.15.98:0/2182796287 conn(0x7f9e740fa950 msgr2=0x7f9e740fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 -- 172.21.15.98:0/2182796287 shutdown_connections 2024-03-18T20:25:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.401+0000 7f9e7a06f700 1 -- 172.21.15.98:0/2182796287 wait complete. 2024-03-18T20:25:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 Processor -- start 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 -- start start 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e740ffec0 0x7f9e74196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74101270 0x7f9e7419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419e750 con 0x7f9e74100880 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419e890 con 0x7f9e74101270 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e7a06f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419ea20 con 0x7f9e740ffec0 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34884/0 (socket says 172.21.15.98:34884) 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 -- 172.21.15.98:0/710512341 learned_addr learned my addr 172.21.15.98:0/710512341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e737fe700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e740ffec0 0x7f9e74196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e73fff700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74101270 0x7f9e7419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 -- 172.21.15.98:0/710512341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e740ffec0 msgr2=0x7f9e74196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e740ffec0 0x7f9e74196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 -- 172.21.15.98:0/710512341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74101270 msgr2=0x7f9e7419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74101270 0x7f9e7419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 -- 172.21.15.98:0/710512341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640095f0 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.402+0000 7f9e72ffd700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9e64005740 tx=0x7f9e64009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.403+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6401d070 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.403+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7419eca0 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.403+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9e6400be70 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.403+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64021410 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.403+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7419f0c0 con 0x7f9e74100880 2024-03-18T20:25:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.404+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e74067260 con 0x7f9e74100880 2024-03-18T20:25:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.405+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f9e6400f640 con 0x7f9e74100880 2024-03-18T20:25:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.405+0000 7f9e70ff9700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 0x7f9e5c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.405+0000 7f9e737fe700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 0x7f9e5c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.405+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9e640ce330 con 0x7f9e74100880 2024-03-18T20:25:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.406+0000 7f9e737fe700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 0x7f9e5c0a3670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e74067bd0 tx=0x7f9e6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.409+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e64097800 con 0x7f9e74100880 2024-03-18T20:25:38.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.605+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e7419f3b0 con 0x7f9e74100880 2024-03-18T20:25:38.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.605+0000 7f9e70ff9700 1 -- 172.21.15.98:0/710512341 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e6409ae20 con 0x7f9e74100880 2024-03-18T20:25:38.610 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:25:38.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.607+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 msgr2=0x7f9e5c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.607+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 0x7f9e5c0a3670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e74067bd0 tx=0x7f9e6800b410 comp rx=0 tx=0).stop 2024-03-18T20:25:38.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.607+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 msgr2=0x7f9e74197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.607+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9e64005740 tx=0x7f9e64009580 comp rx=0 tx=0).stop 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.607+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 shutdown_connections 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9e5c0a11b0 0x7f9e5c0a3670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e74101270 0x7f9e7419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e740ffec0 0x7f9e74196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 --2- 172.21.15.98:0/710512341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74100880 0x7f9e74197160 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 >> 172.21.15.98:0/710512341 conn(0x7f9e740fa950 msgr2=0x7f9e740feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 shutdown_connections 2024-03-18T20:25:38.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.608+0000 7f9e7a06f700 1 -- 172.21.15.98:0/710512341 wait complete. 2024-03-18T20:25:38.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:25:38.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 -- 172.21.15.98:0/3172175129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 msgr2=0x7fab60100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 --2- 172.21.15.98:0/3172175129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60100080 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fab500099c0 tx=0x7fab50009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 -- 172.21.15.98:0/3172175129 shutdown_connections 2024-03-18T20:25:38.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 --2- 172.21.15.98:0/3172175129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab601080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 --2- 172.21.15.98:0/3172175129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60100080 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 --2- 172.21.15.98:0/3172175129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 0x7fab601012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 -- 172.21.15.98:0/3172175129 >> 172.21.15.98:0/3172175129 conn(0x7fab600fb1d0 msgr2=0x7fab600fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 -- 172.21.15.98:0/3172175129 shutdown_connections 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.745+0000 7fab680ad700 1 -- 172.21.15.98:0/3172175129 wait complete. 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 Processor -- start 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 -- start start 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 0x7fab60102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab60102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60102f10 con 0x7fab600ffc60 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60103080 con 0x7fab60100e50 2024-03-18T20:25:38.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab680ad700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601a0c30 con 0x7fab601017f0 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65e49700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65e49700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34910/0 (socket says 172.21.15.98:34910) 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65e49700 1 -- 172.21.15.98:0/2925460107 learned_addr learned my addr 172.21.15.98:0/2925460107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab6664a700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab60102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65648700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 0x7fab60102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65e49700 1 -- 172.21.15.98:0/2925460107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 msgr2=0x7fab60102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.746+0000 7fab65e49700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab60102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab65e49700 1 -- 172.21.15.98:0/2925460107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 msgr2=0x7fab60102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab65e49700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 0x7fab60102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab65e49700 1 -- 172.21.15.98:0/2925460107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500095f0 con 0x7fab600ffc60 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab6664a700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab60102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab65e49700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fab500098e0 tx=0x7fab50003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5001d070 con 0x7fab600ffc60 2024-03-18T20:25:38.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab601a0e60 con 0x7fab600ffc60 2024-03-18T20:25:38.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fab5000bd50 con 0x7fab600ffc60 2024-03-18T20:25:38.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50022cf0 con 0x7fab600ffc60 2024-03-18T20:25:38.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.747+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab601a11d0 con 0x7fab600ffc60 2024-03-18T20:25:38.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.748+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60067af0 con 0x7fab600ffc60 2024-03-18T20:25:38.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.749+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fab50022470 con 0x7fab600ffc60 2024-03-18T20:25:38.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.749+0000 7fab56ffd700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 0x7fab4c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.750+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fab500ce8b0 con 0x7fab600ffc60 2024-03-18T20:25:38.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.750+0000 7fab65648700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 0x7fab4c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.750+0000 7fab65648700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 0x7fab4c0a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fab5c005950 tx=0x7fab5c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.754+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fab50097d80 con 0x7fab600ffc60 2024-03-18T20:25:38.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.953+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fab601a13c0 con 0x7fab600ffc60 2024-03-18T20:25:38.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.954+0000 7fab56ffd700 1 -- 172.21.15.98:0/2925460107 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7fab5009b3a0 con 0x7fab600ffc60 2024-03-18T20:25:38.958 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:38.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 msgr2=0x7fab4c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 0x7fab4c0a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fab5c005950 tx=0x7fab5c0058e0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 msgr2=0x7fab60103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fab500098e0 tx=0x7fab50003b80 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 shutdown_connections 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab601017f0 0x7fab60102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600ffc60 0x7fab60103ba0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fab4c0a11b0 0x7fab4c0a3670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 --2- 172.21.15.98:0/2925460107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab60100e50 0x7fab60102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 >> 172.21.15.98:0/2925460107 conn(0x7fab600fb1d0 msgr2=0x7fab60067e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 shutdown_connections 2024-03-18T20:25:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:38.956+0000 7fab680ad700 1 -- 172.21.15.98:0/2925460107 wait complete. 2024-03-18T20:25:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:38 smithi098 ceph-mon[43462]: pgmap v251: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/251774406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3786911127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:38 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/710512341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:38.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:38 smithi136 ceph-mon[48073]: pgmap v251: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:38 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/251774406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:38 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3786911127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:38 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/710512341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 -- 172.21.15.98:0/786379951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c100d60 msgr2=0x7f555c1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 --2- 172.21.15.98:0/786379951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c100d60 0x7f555c1011c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f55500099c0 tx=0x7f5550009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 -- 172.21.15.98:0/786379951 shutdown_connections 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 --2- 172.21.15.98:0/786379951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c101700 0x7f555c107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 --2- 172.21.15.98:0/786379951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 0x7f555c0fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 --2- 172.21.15.98:0/786379951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c100d60 0x7f555c1011c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 -- 172.21.15.98:0/786379951 >> 172.21.15.98:0/786379951 conn(0x7f555c0fb100 msgr2=0x7f555c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 -- 172.21.15.98:0/786379951 shutdown_connections 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.096+0000 7f55613b6700 1 -- 172.21.15.98:0/786379951 wait complete. 2024-03-18T20:25:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 Processor -- start 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 -- start start 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 0x7f555c103ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c101700 0x7f555c102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c102e20 con 0x7f555c101700 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c102f90 con 0x7f555c100d60 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f55613b6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c1a0b40 con 0x7f555c0ffb70 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f555a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f555a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.98:46604/0 (socket says 172.21.15.98:46604) 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f555a7fc700 1 -- 172.21.15.98:0/3719138454 learned_addr learned my addr 172.21.15.98:0/3719138454 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f555affd700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 0x7f555c103ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.097+0000 7f555b7fe700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c101700 0x7f555c102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 -- 172.21.15.98:0/3719138454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 msgr2=0x7f555c103ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 0x7f555c103ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 -- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c101700 msgr2=0x7f555c102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c101700 0x7f555c102680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 -- 172.21.15.98:0/3719138454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500095f0 con 0x7f555c100d60 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f555a7fc700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5550000c00 tx=0x7f555000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555001d070 con 0x7f555c100d60 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c1a0d70 con 0x7f555c100d60 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f555000bd30 con 0x7f555c100d60 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55500173f0 con 0x7f555c100d60 2024-03-18T20:25:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.098+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c1a1130 con 0x7f555c100d60 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.099+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c067e60 con 0x7f555c100d60 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.101+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5550022470 con 0x7f555c100d60 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.101+0000 7f5543fff700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 0x7f55440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.101+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55500cdf50 con 0x7f555c100d60 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.101+0000 7f555affd700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 0x7f55440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.102+0000 7f555affd700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 0x7f55440a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f555c100bc0 tx=0x7f554c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.105+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5550097420 con 0x7f555c100d60 2024-03-18T20:25:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:38 smithi169 ceph-mon[48529]: pgmap v251: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:38 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/251774406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:38 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3786911127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:39.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:38 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/710512341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.368+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f555c1a13b0 con 0x7f555c100d60 2024-03-18T20:25:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.370+0000 7f5543fff700 1 -- 172.21.15.98:0/3719138454 <== mon.2 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7f555009aa40 con 0x7f555c100d60 2024-03-18T20:25:39.373 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:39.374 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 msgr2=0x7f55440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 0x7f55440a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f555c100bc0 tx=0x7f554c009260 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 msgr2=0x7f555c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5550000c00 tx=0x7f555000fa00 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 shutdown_connections 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f555c101700 0x7f555c102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f555c0ffb70 0x7f555c103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f55440a11b0 0x7f55440a3670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 --2- 172.21.15.98:0/3719138454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f555c100d60 0x7f555c102140 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 >> 172.21.15.98:0/3719138454 conn(0x7f555c0fb100 msgr2=0x7f555c069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 shutdown_connections 2024-03-18T20:25:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.372+0000 7f55613b6700 1 -- 172.21.15.98:0/3719138454 wait complete. 2024-03-18T20:25:39.377 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:39.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:39.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 -- 172.21.15.98:0/3477935664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a01016e0 msgr2=0x7f65a0103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3477935664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a01016e0 0x7f65a0103ad0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6594009990 tx=0x7f6594009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 -- 172.21.15.98:0/3477935664 shutdown_connections 2024-03-18T20:25:39.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3477935664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a0106940 0x7f65a0108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3477935664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a01016e0 0x7f65a0103ad0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3477935664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 -- 172.21.15.98:0/3477935664 >> 172.21.15.98:0/3477935664 conn(0x7f65a00fb0d0 msgr2=0x7f65a00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 -- 172.21.15.98:0/3477935664 shutdown_connections 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.514+0000 7f65a70ea700 1 -- 172.21.15.98:0/3477935664 wait complete. 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 Processor -- start 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 -- start start 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 0x7f65a0196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 0x7f65a019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a019e5d0 con 0x7f65a0106940 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a019e710 con 0x7f65a0104010 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a70ea700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a019e8a0 con 0x7f65a01016e0 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f659ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f65a5687700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 0x7f65a019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f659ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.98:46628/0 (socket says 172.21.15.98:46628) 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.515+0000 7f659ffff700 1 -- 172.21.15.98:0/3037107962 learned_addr learned my addr 172.21.15.98:0/3037107962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f65a4e86700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 0x7f65a0196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 -- 172.21.15.98:0/3037107962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 msgr2=0x7f65a0196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 0x7f65a0196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 -- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 msgr2=0x7f65a019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 0x7f65a019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 -- 172.21.15.98:0/3037107962 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940095f0 con 0x7f65a0104010 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f65a5687700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 0x7f65a019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f65a4e86700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 0x7f65a0196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659ffff700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e9a0 tx=0x7f658c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c009790 con 0x7f65a0104010 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a019eb80 con 0x7f65a0104010 2024-03-18T20:25:39.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f658c00cbd0 con 0x7f65a0104010 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c018680 con 0x7f65a0104010 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.516+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a019f0e0 con 0x7f65a0104010 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.517+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a00fcc50 con 0x7f65a0104010 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.519+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f658c00cd40 con 0x7f65a0104010 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.519+0000 7f659dffb700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 0x7f65900a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.519+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f658c014070 con 0x7f65a0104010 2024-03-18T20:25:39.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.519+0000 7f65a4e86700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 0x7f65900a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.520+0000 7f65a4e86700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 0x7f65900a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f659400b670 tx=0x7f6594005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.523+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f658c0966c0 con 0x7f65a0104010 2024-03-18T20:25:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.759+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65a002ccb0 con 0x7f65a0104010 2024-03-18T20:25:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.761+0000 7f659dffb700 1 -- 172.21.15.98:0/3037107962 <== mon.2 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f658c099ce0 con 0x7f65a0104010 2024-03-18T20:25:39.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:39.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 msgr2=0x7f65900a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 0x7f65900a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f659400b670 tx=0x7f6594005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 msgr2=0x7f65a0196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e9a0 tx=0x7f658c00ecb0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 shutdown_connections 2024-03-18T20:25:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65a0106940 0x7f65a019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65a01016e0 0x7f65a0196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f65900a11b0 0x7f65900a3670 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 --2- 172.21.15.98:0/3037107962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a0104010 0x7f65a0196fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 >> 172.21.15.98:0/3037107962 conn(0x7f65a00fb0d0 msgr2=0x7f65a0103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 shutdown_connections 2024-03-18T20:25:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.764+0000 7f65a70ea700 1 -- 172.21.15.98:0/3037107962 wait complete. 2024-03-18T20:25:39.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:25:39.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 -- 172.21.15.98:0/1456360621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc0100f70 msgr2=0x7f5cc0107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/1456360621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc0100f70 0x7f5cc0107820 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b0a0 tx=0x7f5cb800b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 -- 172.21.15.98:0/1456360621 shutdown_connections 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/1456360621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc0100f70 0x7f5cc0107820 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/1456360621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cc00ff3e0 0x7f5cc00ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/1456360621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 -- 172.21.15.98:0/1456360621 >> 172.21.15.98:0/1456360621 conn(0x7f5cc00fa950 msgr2=0x7f5cc00fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 -- 172.21.15.98:0/1456360621 shutdown_connections 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.927+0000 7f5cc6d18700 1 -- 172.21.15.98:0/1456360621 wait complete. 2024-03-18T20:25:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 Processor -- start 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- start start 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc00ff3e0 0x7f5cc0196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cc0100f70 0x7f5cc019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc019e830 con 0x7f5cc00ff3e0 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc019e9a0 con 0x7f5cc01005d0 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc019eb60 con 0x7f5cc0100f70 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.98:46650/0 (socket says 172.21.15.98:46650) 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 -- 172.21.15.98:0/3761646248 learned_addr learned my addr 172.21.15.98:0/3761646248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 -- 172.21.15.98:0/3761646248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cc0100f70 msgr2=0x7f5cc019e260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cc0100f70 0x7f5cc019e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 -- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc00ff3e0 msgr2=0x7f5cc0196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc00ff3e0 0x7f5cc0196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 -- 172.21.15.98:0/3761646248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8009d00 con 0x7f5cc01005d0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbffff700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00e9a0 tx=0x7f5cac00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00cbd0 con 0x7f5cc01005d0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5cac00cd30 con 0x7f5cc01005d0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc019ee40 con 0x7f5cc01005d0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac018680 con 0x7f5cc01005d0 2024-03-18T20:25:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.928+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc019f320 con 0x7f5cc01005d0 2024-03-18T20:25:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.929+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc0106880 con 0x7f5cc01005d0 2024-03-18T20:25:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.931+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5cac010430 con 0x7f5cc01005d0 2024-03-18T20:25:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.932+0000 7f5cbdffb700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 0x7f5cb00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.932+0000 7f5cc4ab4700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 0x7f5cb00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.932+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5cac014070 con 0x7f5cc01005d0 2024-03-18T20:25:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.932+0000 7f5cc4ab4700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 0x7f5cb00a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5cc01981c0 tx=0x7f5cb40096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:39.936+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5cac0964d0 con 0x7f5cc01005d0 2024-03-18T20:25:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:39 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:25:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:39 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2925460107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:39.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:39 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3719138454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:39 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:39 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2925460107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:40.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:39 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3719138454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:39 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:39 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2925460107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:40.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:39 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3719138454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.149+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5cc002cee0 con 0x7f5cc01005d0 2024-03-18T20:25:40.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.150+0000 7f5cbdffb700 1 -- 172.21.15.98:0/3761646248 <== mon.2 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5cac099af0 con 0x7f5cc01005d0 2024-03-18T20:25:40.154 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 msgr2=0x7f5cb00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 0x7f5cb00a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5cc01981c0 tx=0x7f5cb40096c0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 msgr2=0x7f5cc0197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00e9a0 tx=0x7f5cac00ed60 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 shutdown_connections 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5cc0100f70 0x7f5cc019e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cc00ff3e0 0x7f5cc0196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5cb00a11b0 0x7f5cb00a3670 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 --2- 172.21.15.98:0/3761646248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5cc01005d0 0x7f5cc0197280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 >> 172.21.15.98:0/3761646248 conn(0x7f5cc00fa950 msgr2=0x7f5cc01039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 shutdown_connections 2024-03-18T20:25:40.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.152+0000 7f5cc6d18700 1 -- 172.21.15.98:0/3761646248 wait complete. 2024-03-18T20:25:40.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 -- 172.21.15.98:0/920893443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 msgr2=0x7f7680106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 --2- 172.21.15.98:0/920893443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f7680106500 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7670009990 tx=0x7f7670009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 -- 172.21.15.98:0/920893443 shutdown_connections 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 --2- 172.21.15.98:0/920893443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7680106a40 0x7f7680108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 --2- 172.21.15.98:0/920893443 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76801017e0 0x7f7680103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 --2- 172.21.15.98:0/920893443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f7680106500 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 -- 172.21.15.98:0/920893443 >> 172.21.15.98:0/920893443 conn(0x7f76800fb1d0 msgr2=0x7f76800fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 -- 172.21.15.98:0/920893443 shutdown_connections 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 -- 172.21.15.98:0/920893443 wait complete. 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.313+0000 7f76886d2700 1 Processor -- start 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 -- start start 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76801017e0 0x7f7680101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 0x7f76800ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7680100470 con 0x7f7680104110 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801005e0 con 0x7f76801017e0 2024-03-18T20:25:40.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76886d2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801a0c40 con 0x7f7680106a40 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34992/0 (socket says 172.21.15.98:34992) 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 -- 172.21.15.98:0/272373858 learned_addr learned my addr 172.21.15.98:0/272373858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f768646e700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76801017e0 0x7f7680101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7686c6f700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 0x7f76800ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 -- 172.21.15.98:0/272373858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 msgr2=0x7f76800ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 0x7f76800ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 -- 172.21.15.98:0/272373858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76801017e0 msgr2=0x7f7680101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76801017e0 0x7f7680101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 -- 172.21.15.98:0/272373858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700095f0 con 0x7f7680104110 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7686c6f700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 0x7f76800ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f7685c6d700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7670004710 tx=0x7f7670009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.314+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767001d070 con 0x7f7680104110 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.315+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76801a0e70 con 0x7f7680104110 2024-03-18T20:25:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.315+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f767000be70 con 0x7f7680104110 2024-03-18T20:25:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.315+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670021410 con 0x7f7680104110 2024-03-18T20:25:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.315+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801a1230 con 0x7f7680104110 2024-03-18T20:25:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.315+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76800fcd50 con 0x7f7680104110 2024-03-18T20:25:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.317+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f767000f6f0 con 0x7f7680104110 2024-03-18T20:25:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.318+0000 7f76777fe700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 0x7f766c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.318+0000 7f768646e700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 0x7f766c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.318+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f76700ce2b0 con 0x7f7680104110 2024-03-18T20:25:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.318+0000 7f768646e700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 0x7f766c0a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f767c005950 tx=0x7f767c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.321+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7670097780 con 0x7f7680104110 2024-03-18T20:25:40.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.526+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7680067af0 con 0x7f7680104110 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.526+0000 7f76777fe700 1 -- 172.21.15.98:0/272373858 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f767009ada0 con 0x7f7680104110 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.528+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 msgr2=0x7f766c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.528+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 0x7f766c0a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f767c005950 tx=0x7f767c00b410 comp rx=0 tx=0).stop 2024-03-18T20:25:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.528+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 msgr2=0x7f76800ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.528+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7670004710 tx=0x7f7670009580 comp rx=0 tx=0).stop 2024-03-18T20:25:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 shutdown_connections 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7680106a40 0x7f76800ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f766c0a11b0 0x7f766c0a3670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76801017e0 0x7f7680101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 --2- 172.21.15.98:0/272373858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7680104110 0x7f76800ff790 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 >> 172.21.15.98:0/272373858 conn(0x7f76800fb1d0 msgr2=0x7f7680103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 shutdown_connections 2024-03-18T20:25:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.529+0000 7f76886d2700 1 -- 172.21.15.98:0/272373858 wait complete. 2024-03-18T20:25:40.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 -- 172.21.15.98:0/884089552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 msgr2=0x7fef900ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 --2- 172.21.15.98:0/884089552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef900ff480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fef80009990 tx=0x7fef80009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 -- 172.21.15.98:0/884089552 shutdown_connections 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 --2- 172.21.15.98:0/884089552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef90102a70 0x7fef90109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 --2- 172.21.15.98:0/884089552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef900ff480 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 --2- 172.21.15.98:0/884089552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef901020d0 0x7fef90102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 -- 172.21.15.98:0/884089552 >> 172.21.15.98:0/884089552 conn(0x7fef900fa950 msgr2=0x7fef900fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 -- 172.21.15.98:0/884089552 shutdown_connections 2024-03-18T20:25:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.668+0000 7fef957b4700 1 -- 172.21.15.98:0/884089552 wait complete. 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 Processor -- start 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 -- start start 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef901020d0 0x7fef90197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef90102a70 0x7fef9019e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9019e750 con 0x7fef900ff060 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9019e890 con 0x7fef901020d0 2024-03-18T20:25:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef957b4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9019ea20 con 0x7fef90102a70 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35010/0 (socket says 172.21.15.98:35010) 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 -- 172.21.15.98:0/450430560 learned_addr learned my addr 172.21.15.98:0/450430560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8f7fe700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef90102a70 0x7fef9019e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8e7fc700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef901020d0 0x7fef90197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 -- 172.21.15.98:0/450430560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef90102a70 msgr2=0x7fef9019e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef90102a70 0x7fef9019e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 -- 172.21.15.98:0/450430560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef901020d0 msgr2=0x7fef90197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.669+0000 7fef8effd700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef901020d0 0x7fef90197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.670+0000 7fef8effd700 1 -- 172.21.15.98:0/450430560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef800095f0 con 0x7fef900ff060 2024-03-18T20:25:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.670+0000 7fef8effd700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef80003f60 tx=0x7fef80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.673+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8001d070 con 0x7fef900ff060 2024-03-18T20:25:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.673+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef8000bdb0 con 0x7fef900ff060 2024-03-18T20:25:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.673+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9019eca0 con 0x7fef900ff060 2024-03-18T20:25:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.673+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80022dd0 con 0x7fef900ff060 2024-03-18T20:25:40.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.673+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9019f120 con 0x7fef900ff060 2024-03-18T20:25:40.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.674+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef901083a0 con 0x7fef900ff060 2024-03-18T20:25:40.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.675+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fef80022570 con 0x7fef900ff060 2024-03-18T20:25:40.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.675+0000 7fef77fff700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 0x7fef780a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.675+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fef800ce0a0 con 0x7fef900ff060 2024-03-18T20:25:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.675+0000 7fef8e7fc700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 0x7fef780a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.676+0000 7fef8e7fc700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 0x7fef780a36f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fef84009bf0 tx=0x7fef84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.680+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef80097570 con 0x7fef900ff060 2024-03-18T20:25:40.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: pgmap v252: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:40.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3037107962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3761646248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:40 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/272373858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.875+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef9002cdc0 con 0x7fef900ff060 2024-03-18T20:25:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.876+0000 7fef77fff700 1 -- 172.21.15.98:0/450430560 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef8009ab90 con 0x7fef900ff060 2024-03-18T20:25:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 msgr2=0x7fef780a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 0x7fef780a36f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fef84009bf0 tx=0x7fef84009260 comp rx=0 tx=0).stop 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 msgr2=0x7fef90196c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fef80003f60 tx=0x7fef80004730 comp rx=0 tx=0).stop 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 shutdown_connections 2024-03-18T20:25:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef90102a70 0x7fef9019e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef900ff060 0x7fef90196c20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fef780a1230 0x7fef780a36f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 --2- 172.21.15.98:0/450430560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef901020d0 0x7fef90197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 >> 172.21.15.98:0/450430560 conn(0x7fef900fa950 msgr2=0x7fef901054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 shutdown_connections 2024-03-18T20:25:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:40.878+0000 7fef957b4700 1 -- 172.21.15.98:0/450430560 wait complete. 2024-03-18T20:25:40.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:25:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: pgmap v252: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3037107962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3761646248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:25:40.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:25:40.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:40.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:40 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/272373858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 -- 172.21.15.98:0/1552438096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 msgr2=0x7f6798108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 --2- 172.21.15.98:0/1552438096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f6798108ec0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f679000b0a0 tx=0x7f679000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 -- 172.21.15.98:0/1552438096 shutdown_connections 2024-03-18T20:25:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 --2- 172.21.15.98:0/1552438096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f6798108ec0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 --2- 172.21.15.98:0/1552438096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 0x7f6798103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 --2- 172.21.15.98:0/1552438096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 0x7f6798106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 -- 172.21.15.98:0/1552438096 >> 172.21.15.98:0/1552438096 conn(0x7f67980fb200 msgr2=0x7f67980fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 -- 172.21.15.98:0/1552438096 shutdown_connections 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.016+0000 7f67a0636700 1 -- 172.21.15.98:0/1552438096 wait complete. 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 Processor -- start 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 -- start start 2024-03-18T20:25:41.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 0x7f6798196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 0x7f6798197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679819e800 con 0x7f6798106a50 2024-03-18T20:25:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679819e970 con 0x7f67981017f0 2024-03-18T20:25:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f67a0636700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679819eb30 con 0x7f6798104120 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35024/0 (socket says 172.21.15.98:35024) 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 -- 172.21.15.98:0/3116016570 learned_addr learned my addr 172.21.15.98:0/3116016570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679dbd1700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 0x7f6798197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679e3d2700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 0x7f6798196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 -- 172.21.15.98:0/3116016570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 msgr2=0x7f6798197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 0x7f6798197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 -- 172.21.15.98:0/3116016570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 msgr2=0x7f6798196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 0x7f6798196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.017+0000 7f679ebd3700 1 -- 172.21.15.98:0/3116016570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790009d00 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f679ebd3700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6790007970 tx=0x7f6790004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679000e040 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6790004e10 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679819edb0 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679001b9a0 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.018+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679819f1d0 con 0x7f6798106a50 2024-03-18T20:25:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.019+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67980fcd80 con 0x7f6798106a50 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.020+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f679001dcb0 con 0x7f6798106a50 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.020+0000 7f678f7fe700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 0x7f67840a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.021+0000 7f679e3d2700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 0x7f67840a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.021+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f67900cf2f0 con 0x7f6798106a50 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.021+0000 7f679e3d2700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 0x7f67840a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6788005950 tx=0x7f6788005d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.024+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f67900987c0 con 0x7f6798106a50 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: pgmap v252: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3037107962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3761646248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:25:41.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:25:41.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:25:41.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:40 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/272373858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.223+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f679802cdc0 con 0x7f6798106a50 2024-03-18T20:25:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.223+0000 7f678f7fe700 1 -- 172.21.15.98:0/3116016570 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f679009bde0 con 0x7f6798106a50 2024-03-18T20:25:41.227 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.225+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 msgr2=0x7f67840a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.225+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 0x7f67840a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6788005950 tx=0x7f6788005d10 comp rx=0 tx=0).stop 2024-03-18T20:25:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 msgr2=0x7f679819e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6790007970 tx=0x7f6790004b00 comp rx=0 tx=0).stop 2024-03-18T20:25:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 shutdown_connections 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798106a50 0x7f679819e230 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f67840a11b0 0x7f67840a3670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67981017f0 0x7f6798196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 --2- 172.21.15.98:0/3116016570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6798104120 0x7f6798197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 >> 172.21.15.98:0/3116016570 conn(0x7f67980fb200 msgr2=0x7f6798102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 shutdown_connections 2024-03-18T20:25:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.226+0000 7f67a0636700 1 -- 172.21.15.98:0/3116016570 wait complete. 2024-03-18T20:25:41.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:41.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 -- 172.21.15.98:0/2510337641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 msgr2=0x7fc38c1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 --2- 172.21.15.98:0/2510337641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c1010a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0099c0 tx=0x7fc37c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 -- 172.21.15.98:0/2510337641 shutdown_connections 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 --2- 172.21.15.98:0/2510337641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c102810 0x7fc38c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 --2- 172.21.15.98:0/2510337641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c1010a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 --2- 172.21.15.98:0/2510337641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c101e70 0x7fc38c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 -- 172.21.15.98:0/2510337641 >> 172.21.15.98:0/2510337641 conn(0x7fc38c0fc1f0 msgr2=0x7fc38c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.360+0000 7fc3924db700 1 -- 172.21.15.98:0/2510337641 shutdown_connections 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 -- 172.21.15.98:0/2510337641 wait complete. 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 Processor -- start 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 -- start start 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c101e70 0x7fc38c197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 0x7fc38c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c19e6a0 con 0x7fc38c100c80 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c19e7e0 con 0x7fc38c101e70 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.361+0000 7fc3924db700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c19e970 con 0x7fc38c102810 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35038/0 (socket says 172.21.15.98:35038) 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 -- 172.21.15.98:0/1625403316 learned_addr learned my addr 172.21.15.98:0/1625403316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38b7fe700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c101e70 0x7fc38c197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc390a78700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 0x7fc38c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 -- 172.21.15.98:0/1625403316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 msgr2=0x7fc38c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 0x7fc38c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 -- 172.21.15.98:0/1625403316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c101e70 msgr2=0x7fc38c197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c101e70 0x7fc38c197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 -- 172.21.15.98:0/1625403316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0095f0 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc390a78700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 0x7fc38c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc38bfff700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009fd0 tx=0x7fc37c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c01d070 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c19ebf0 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc37c00bd10 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c004ba0 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.362+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c19f070 con 0x7fc38c100c80 2024-03-18T20:25:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.363+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c1081f0 con 0x7fc38c100c80 2024-03-18T20:25:41.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.364+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc37c004d00 con 0x7fc38c100c80 2024-03-18T20:25:41.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.365+0000 7fc3897fa700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 0x7fc3740a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.365+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc37c0cde70 con 0x7fc38c100c80 2024-03-18T20:25:41.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.365+0000 7fc38b7fe700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 0x7fc3740a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.365+0000 7fc38b7fe700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 0x7fc3740a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc380007c60 tx=0x7fc3800073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.368+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc37c0972e0 con 0x7fc38c100c80 2024-03-18T20:25:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.618+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc38c067a10 con 0x7fc38c100c80 2024-03-18T20:25:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.619+0000 7fc3897fa700 1 -- 172.21.15.98:0/1625403316 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7fc37c09a900 con 0x7fc38c100c80 2024-03-18T20:25:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:41.623 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:41.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 msgr2=0x7fc3740a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 0x7fc3740a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc380007c60 tx=0x7fc3800073d0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 msgr2=0x7fc38c196bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009fd0 tx=0x7fc37c00fa00 comp rx=0 tx=0).stop 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 shutdown_connections 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102810 0x7fc38c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c100c80 0x7fc38c196bc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc3740a1230 0x7fc3740a36f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 --2- 172.21.15.98:0/1625403316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc38c101e70 0x7fc38c197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 >> 172.21.15.98:0/1625403316 conn(0x7fc38c0fc1f0 msgr2=0x7fc38c105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 shutdown_connections 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.621+0000 7fc3924db700 1 -- 172.21.15.98:0/1625403316 wait complete. 2024-03-18T20:25:41.626 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:41.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2446597519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 msgr2=0x7f2fe00fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2446597519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe00fff80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00099c0 tx=0x7f2fd0009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2446597519 shutdown_connections 2024-03-18T20:25:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2446597519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fe01016f0 0x7f2fe0107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2446597519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe00fff80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2446597519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe0100d50 0x7f2fe01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2446597519 >> 172.21.15.98:0/2446597519 conn(0x7f2fe00fb0d0 msgr2=0x7f2fe00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2446597519 shutdown_connections 2024-03-18T20:25:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.756+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2446597519 wait complete. 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.757+0000 7f2fe49a0700 1 Processor -- start 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.757+0000 7f2fe49a0700 1 -- start start 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.757+0000 7f2fe49a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe0100d50 0x7f2fe0195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fe01016f0 0x7f2fe019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe019d560 con 0x7f2fe00ffb60 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe019d6a0 con 0x7f2fe0100d50 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe019d830 con 0x7f2fe01016f0 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35056/0 (socket says 172.21.15.98:35056) 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 -- 172.21.15.98:0/2344906117 learned_addr learned my addr 172.21.15.98:0/2344906117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 -- 172.21.15.98:0/2344906117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fe01016f0 msgr2=0x7f2fe019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fe01016f0 0x7f2fe019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 -- 172.21.15.98:0/2344906117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe0100d50 msgr2=0x7f2fe0195fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe0100d50 0x7f2fe0195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 -- 172.21.15.98:0/2344906117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00095f0 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fde59c700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009f50 tx=0x7f2fd0005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd001d070 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2fd000bda0 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe019dab0 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0022d20 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.758+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe019df60 con 0x7f2fe00ffb60 2024-03-18T20:25:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.759+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe0107070 con 0x7f2fe00ffb60 2024-03-18T20:25:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.760+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f2fd0003f80 con 0x7f2fe00ffb60 2024-03-18T20:25:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.761+0000 7f2fd77fe700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 0x7f2fcc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.761+0000 7f2fddd9b700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 0x7f2fcc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.761+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2fd00cdfd0 con 0x7f2fe00ffb60 2024-03-18T20:25:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.761+0000 7f2fddd9b700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 0x7f2fcc0a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800a990 tx=0x7f2fc8005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.765+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2fd0097440 con 0x7f2fe00ffb60 2024-03-18T20:25:41.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.972+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fe0067a10 con 0x7f2fe00ffb60 2024-03-18T20:25:41.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.973+0000 7f2fd77fe700 1 -- 172.21.15.98:0/2344906117 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23960 (secure 0 0 0) 0x7f2fd009aa60 con 0x7f2fe00ffb60 2024-03-18T20:25:41.977 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:41.978 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:24:45.850041+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:41.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 msgr2=0x7f2fcc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 0x7f2fcc0a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800a990 tx=0x7f2fc8005c80 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 msgr2=0x7f2fe0195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009f50 tx=0x7f2fd0005b00 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 shutdown_connections 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fe01016f0 0x7f2fe019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fe00ffb60 0x7f2fe0195a80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2fcc0a11b0 0x7f2fcc0a3670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 --2- 172.21.15.98:0/2344906117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe0100d50 0x7f2fe0195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 >> 172.21.15.98:0/2344906117 conn(0x7f2fe00fb0d0 msgr2=0x7f2fe01041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 shutdown_connections 2024-03-18T20:25:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:41.976+0000 7f2fe49a0700 1 -- 172.21.15.98:0/2344906117 wait complete. 2024-03-18T20:25:41.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/450430560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3116016570' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:41 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1625403316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/450430560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3116016570' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:41.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:41 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1625403316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:41.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 -- 172.21.15.98:0/1658173854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100510 msgr2=0x7f4fe0100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1658173854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100510 0x7f4fe0100970 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000aad0 tx=0x7f4fd000ade0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 -- 172.21.15.98:0/1658173854 shutdown_connections 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1658173854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe0100eb0 0x7f4fe0107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1658173854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe00ff320 0x7f4fe00ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1658173854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100510 0x7f4fe0100970 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 -- 172.21.15.98:0/1658173854 >> 172.21.15.98:0/1658173854 conn(0x7f4fe00fa890 msgr2=0x7f4fe00fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 -- 172.21.15.98:0/1658173854 shutdown_connections 2024-03-18T20:25:42.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.111+0000 7f4fe533a700 1 -- 172.21.15.98:0/1658173854 wait complete. 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 Processor -- start 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 -- start start 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe00ff320 0x7f4fe01948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 0x7f4fe0194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe019c420 con 0x7f4fe0100eb0 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe019c560 con 0x7f4fe00ff320 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fe533a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe019c6f0 con 0x7f4fe0100510 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fdf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fdf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35074/0 (socket says 172.21.15.98:35074) 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fdf7fe700 1 -- 172.21.15.98:0/1922838493 learned_addr learned my addr 172.21.15.98:0/1922838493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.112+0000 7f4fdeffd700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe00ff320 0x7f4fe01948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fde7fc700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 0x7f4fe0194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 -- 172.21.15.98:0/1922838493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 msgr2=0x7f4fe0194e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 0x7f4fe0194e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 -- 172.21.15.98:0/1922838493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe00ff320 msgr2=0x7f4fe01948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe00ff320 0x7f4fe01948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 -- 172.21.15.98:0/1922838493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd000a700 con 0x7f4fe0100eb0 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fde7fc700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 0x7f4fe0194e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fdf7fe700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007ee0 tx=0x7f4fd800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd800ddd0 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe019c9d0 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4fd800f040 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8004870 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.113+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe019cee0 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.114+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe01067f0 con 0x7f4fe0100eb0 2024-03-18T20:25:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.115+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4fd8003a00 con 0x7f4fe0100eb0 2024-03-18T20:25:42.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.115+0000 7f4fd7fff700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 0x7f4fcc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.115+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4fd80ccde0 con 0x7f4fe0100eb0 2024-03-18T20:25:42.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.116+0000 7f4fdeffd700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 0x7f4fcc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.116+0000 7f4fdeffd700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 0x7f4fcc0a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8005950 tx=0x7f4fc80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.119+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4fd8096220 con 0x7f4fe0100eb0 2024-03-18T20:25:42.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/450430560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:42.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3116016570' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:42.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:41 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1625403316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.362+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fe002d150 con 0x7f4fe0100eb0 2024-03-18T20:25:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.363+0000 7f4fd7fff700 1 -- 172.21.15.98:0/1922838493 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7f4fd8023090 con 0x7f4fe0100eb0 2024-03-18T20:25:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:42.367 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 msgr2=0x7f4fcc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 0x7f4fcc0a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8005950 tx=0x7f4fc80058e0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 msgr2=0x7f4fe019be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007ee0 tx=0x7f4fd800d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 shutdown_connections 2024-03-18T20:25:42.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fe0100eb0 0x7f4fe019be50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4fcc0a11b0 0x7f4fcc0a3670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fe00ff320 0x7f4fe01948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 --2- 172.21.15.98:0/1922838493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4fe0100510 0x7f4fe0194e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 >> 172.21.15.98:0/1922838493 conn(0x7f4fe00fa890 msgr2=0x7f4fe0103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 shutdown_connections 2024-03-18T20:25:42.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.365+0000 7f4fe533a700 1 -- 172.21.15.98:0/1922838493 wait complete. 2024-03-18T20:25:42.371 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:42.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T20:24:41.335739+0000', 'modified': '2024-03-18T20:25:00.478029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24494, 'mds_1': 24475, 'mds_2': 24461}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6835/4136937912', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 4136937912}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 4136937912}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6835/477760816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 477760816}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 477760816}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6837/1340440313', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6836', 'nonce': 1340440313}, {'type': 'v1', 'addr': '172.21.15.169:6837', 'nonce': 1340440313}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.136:6837/3347718998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3347718998}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3347718998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.169:6839/2458510608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6838', 'nonce': 2458510608}, {'type': 'v1', 'addr': '172.21.15.169:6839', 'nonce': 2458510608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.136:6839/2871006308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 2871006308}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 2871006308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:25:42.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T20:25:42.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T20:25:42.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 -- 172.21.15.98:0/4248836968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c0ff030 msgr2=0x7f4a9c0ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/4248836968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c0ff450 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009990 tx=0x7f4a84009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 -- 172.21.15.98:0/4248836968 shutdown_connections 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/4248836968 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/4248836968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c0ff450 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/4248836968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c102000 0x7f4a9c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 -- 172.21.15.98:0/4248836968 >> 172.21.15.98:0/4248836968 conn(0x7f4a9c0fa960 msgr2=0x7f4a9c0fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 -- 172.21.15.98:0/4248836968 shutdown_connections 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 -- 172.21.15.98:0/4248836968 wait complete. 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.504+0000 7f4aa1daf700 1 Processor -- start 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 -- start start 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c102000 0x7f4a9c197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19e740 con 0x7f4a9c1029a0 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19e880 con 0x7f4a9c0ff030 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4aa1daf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19ea10 con 0x7f4a9c102000 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35094/0 (socket says 172.21.15.98:35094) 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 -- 172.21.15.98:0/423732712 learned_addr learned my addr 172.21.15.98:0/423732712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9affd700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c102000 0x7f4a9c197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 -- 172.21.15.98:0/423732712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c102000 msgr2=0x7f4a9c197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c102000 0x7f4a9c197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 -- 172.21.15.98:0/423732712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c0ff030 msgr2=0x7f4a9c196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 -- 172.21.15.98:0/423732712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a840095f0 con 0x7f4a9c1029a0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.505+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400b970 tx=0x7f4a9400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.506+0000 7f4a98ff9700 1 -- 172.21.15.98:0/423732712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9400d860 con 0x7f4a9c1029a0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.506+0000 7f4a98ff9700 1 -- 172.21.15.98:0/423732712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a9400de50 con 0x7f4a9c1029a0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.506+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c19ecf0 con 0x7f4a9c1029a0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.506+0000 7f4a98ff9700 1 -- 172.21.15.98:0/423732712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94017400 con 0x7f4a9c1029a0 2024-03-18T20:25:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.506+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c19f1b0 con 0x7f4a9c1029a0 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.508+0000 7f4a98ff9700 1 -- 172.21.15.98:0/423732712 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4a9400d9c0 con 0x7f4a9c1029a0 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.508+0000 7f4a98ff9700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 0x7f4a880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.508+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 0x7f4a880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.508+0000 7f4a98ff9700 1 -- 172.21.15.98:0/423732712 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4a94006e40 con 0x7f4a9c1029a0 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 0x7f4a880a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 Processor -- start 2024-03-18T20:25:42.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 -- start start 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a1c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a1dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088970940 con 0x559088a1c940 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55908896e300 con 0x559088a1bfe0 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.509+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088969bf0 con 0x559088a1d7e0 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35110/0 (socket says 172.21.15.98:35110) 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 -- 172.21.15.98:0/3943100359 learned_addr learned my addr 172.21.15.98:0/3943100359 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a1c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 -- 172.21.15.98:0/3943100359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 msgr2=0x559088a1dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a1dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 -- 172.21.15.98:0/3943100359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 msgr2=0x559088a1c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a1c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 -- 172.21.15.98:0/3943100359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590887eb360 con 0x559088a1c940 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ad70 tx=0x7f4a8c00f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2fd609b45c641a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a91ffb700 1 -- 172.21.15.98:0/3943100359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c00a870 con 0x559088a1c940 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a91ffb700 1 -- 172.21.15.98:0/3943100359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a8c00fec0 con 0x559088a1c940 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.510+0000 7f4a91ffb700 1 -- 172.21.15.98:0/3943100359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c018640 con 0x559088a1c940 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3943100359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 msgr2=0x559088a1d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ad70 tx=0x7f4a8c00f6a0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3943100359 shutdown_connections 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a1dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a1c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3943100359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a1d200 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3943100359 >> 172.21.15.98:0/3943100359 conn(0x55908898d1d0 msgr2=0x559088a12b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3943100359 shutdown_connections 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3943100359 wait complete. 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 -- start start 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.511+0000 7f4aadf44b80 1 Processor -- start 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a25ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1c940 0x559088a26420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088905d30 con 0x559088a1d7e0 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590889059d0 con 0x559088a1bfe0 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590887ed650 con 0x559088a1c940 2024-03-18T20:25:42.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 10 client.0 _ll_register_callbacks cb 0x559088771350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35116/0 (socket says 172.21.15.98:35116) 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 -- 172.21.15.98:0/2024497740 learned_addr learned my addr 172.21.15.98:0/2024497740 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a25ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9affd700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1c940 0x559088a26420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 -- 172.21.15.98:0/2024497740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1c940 msgr2=0x559088a26420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1c940 0x559088a26420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 -- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 msgr2=0x559088a25ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a25ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 -- 172.21.15.98:0/2024497740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c00a9d0 con 0x559088a1d7e0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a25ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a94095ea0 tx=0x7f4a94099430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1af943415a5b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a7affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9409a0a0 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4a9400e430 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.512+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590889626b0 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a940999a0 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 10 client.15183 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559088797b10 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5590889626b0 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4a940b2200 con 0x559088a1d7e0 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 10 client.15183 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 10 client.15183 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 20 client.15183 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a30d80 0x559088a360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 4 client.15183 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 --> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590887b8ca0 con 0x559088a30d80 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.513+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a30d80 0x559088a360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.514+0000 7f4a7affd700 10 client.15183 ms_handle_connect on v2:172.21.15.136:6836/3347718998 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.514+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a30d80 0x559088a360a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c02374b82efd3ac3 server_cookie=8287eebaca76c2d8 in_seq=0 out_seq=0 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4a7affd700 1 -- 172.21.15.98:0/2024497740 <== mds.2 v2:172.21.15.136:6836/3347718998 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5590887b8ca0 con 0x559088a30d80 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4a7affd700 10 client.15183 handle_command_reply: tid=0 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 client.15183 shutdown 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 msgr2=0x559088a2d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a94095ea0 tx=0x7f4a94099430 comp rx=0 tx=0).stop 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 shutdown_connections 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a30d80 0x559088a360a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a2d410 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a25ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/2024497740 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1c940 0x559088a26420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4a7affd700 0 client.15183 ms_handle_reset on v2:172.21.15.136:6836/3347718998 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 >> 172.21.15.98:0/2024497740 conn(0x55908898d1d0 msgr2=0x559088a121a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 shutdown_connections 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 1 -- 172.21.15.98:0/2024497740 wait complete. 2024-03-18T20:25:42.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.515+0000 7f4aadf44b80 20 client.15183 trim_cache size 0 max 16384 2024-03-18T20:25:42.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.532+0000 7f4aadf44b80 1 Processor -- start 2024-03-18T20:25:42.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.532+0000 7f4aadf44b80 1 -- start start 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1bfe0 0x559088a1dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1c940 0x559088a2d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590889626b0 con 0x559088a1d7e0 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088927ad0 con 0x559088a1c940 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088962990 con 0x559088a1bfe0 2024-03-18T20:25:42.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35130/0 (socket says 172.21.15.98:35130) 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 -- 172.21.15.98:0/1202082720 learned_addr learned my addr 172.21.15.98:0/1202082720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 -- 172.21.15.98:0/1202082720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1bfe0 msgr2=0x559088a1dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1bfe0 0x559088a1dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 -- 172.21.15.98:0/1202082720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1c940 msgr2=0x559088a2d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1c940 0x559088a2d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 -- 172.21.15.98:0/1202082720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55908892fe50 con 0x559088a1d7e0 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a9bfff700 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400b940 tx=0x7f4a94004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08b4c70b0645be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a927fc700 1 -- 172.21.15.98:0/1202082720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9409a330 con 0x559088a1d7e0 2024-03-18T20:25:42.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a927fc700 1 -- 172.21.15.98:0/1202082720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a9400e430 con 0x559088a1d7e0 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.533+0000 7f4a927fc700 1 -- 172.21.15.98:0/1202082720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94099a10 con 0x559088a1d7e0 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- 172.21.15.98:0/1202082720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 msgr2=0x559088a12ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400b940 tx=0x7f4a94004be0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- 172.21.15.98:0/1202082720 shutdown_connections 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1d7e0 0x559088a12ad0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1bfe0 0x559088a1dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/1202082720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1c940 0x559088a2d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- 172.21.15.98:0/1202082720 >> 172.21.15.98:0/1202082720 conn(0x55908898d1d0 msgr2=0x559088a23d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- 172.21.15.98:0/1202082720 shutdown_connections 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- 172.21.15.98:0/1202082720 wait complete. 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 -- start start 2024-03-18T20:25:42.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.534+0000 7f4aadf44b80 1 Processor -- start 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a29900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088a2ce80 con 0x559088a1c940 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088a2cff0 con 0x559088a1bfe0 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559088a2d1b0 con 0x559088a1d7e0 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 10 client.0 _ll_register_callbacks cb 0x7f4a94099770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35138/0 (socket says 172.21.15.98:35138) 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 -- 172.21.15.98:0/3616074589 learned_addr learned my addr 172.21.15.98:0/3616074589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a29900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 -- 172.21.15.98:0/3616074589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 msgr2=0x559088a2c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 -- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 msgr2=0x559088a29900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a29900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 -- 172.21.15.98:0/3616074589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559088797b10 con 0x559088a1c940 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a29900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a9affd700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00a3e0 tx=0x7f4a8c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f74bb0b537762ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a7affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c00fec0 con 0x559088a1c940 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4a8c005df0 con 0x559088a1c940 2024-03-18T20:25:42.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559088a0aa80 con 0x559088a1c940 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a8c00a850 con 0x559088a1c940 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 10 client.15189 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.535+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5590887efc80 con 0x559088a1c940 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x559088a0aa80 con 0x559088a1c940 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4a8c003ee0 con 0x559088a1c940 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 10 client.15189 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 10 client.15189 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 20 client.15189 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a89d30 0x559088a866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 4 client.15189 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 --> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5590887edc90 con 0x559088a89d30 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a89d30 0x559088a866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4a7affd700 10 client.15189 ms_handle_connect on v2:172.21.15.136:6836/3347718998 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.536+0000 7f4a9b7fe700 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a89d30 0x559088a866d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4c63de372fcb901 server_cookie=c61448b22313751f in_seq=0 out_seq=0 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4a7affd700 1 -- 172.21.15.98:0/3616074589 <== mds.2 v2:172.21.15.136:6836/3347718998 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5590887edc90 con 0x559088a89d30 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4a7affd700 10 client.15189 handle_command_reply: tid=0 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 client.15189 shutdown 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 msgr2=0x559088a29e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00a3e0 tx=0x7f4a8c0049f0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 shutdown_connections 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559088a1d7e0 0x559088a2c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559088a1bfe0 0x559088a29900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x559088a89d30 0x559088a866d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 --2- 172.21.15.98:0/3616074589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559088a1c940 0x559088a29e40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4a7affd700 0 client.15189 ms_handle_reset on v2:172.21.15.136:6836/3347718998 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 >> 172.21.15.98:0/3616074589 conn(0x55908898d1d0 msgr2=0x559088a20740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 shutdown_connections 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 1 -- 172.21.15.98:0/3616074589 wait complete. 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aadf44b80 20 client.15189 trim_cache size 0 max 16384 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-03-18T20:25:42.556 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 43.068543259000002, 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 73.766744009000007 2024-03-18T20:25:42.557 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 msgr2=0x7f4a880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 0x7f4a880a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a84005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 msgr2=0x7f4a9c19e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400b970 tx=0x7f4a9400bc80 comp rx=0 tx=0).stop 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 shutdown_connections 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c1029a0 0x7f4a9c19e170 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a880a11b0 0x7f4a880a3670 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a9c0ff030 0x7f4a9c196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 --2- 172.21.15.98:0/423732712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a9c102000 0x7f4a9c197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.537+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 >> 172.21.15.98:0/423732712 conn(0x7f4a9c0fa960 msgr2=0x7f4a9c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.538+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 shutdown_connections 2024-03-18T20:25:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.538+0000 7f4aa1daf700 1 -- 172.21.15.98:0/423732712 wait complete. 2024-03-18T20:25:42.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T20:25:42.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- 172.21.15.98:0/892211203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41017f0 msgr2=0x7f7bc41080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 --2- 172.21.15.98:0/892211203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41017f0 0x7f7bc41080a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00b0d0 tx=0x7f7bbc00b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- 172.21.15.98:0/892211203 shutdown_connections 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 --2- 172.21.15.98:0/892211203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc41017f0 0x7f7bc41080a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 --2- 172.21.15.98:0/892211203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 --2- 172.21.15.98:0/892211203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 0x7f7bc41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- 172.21.15.98:0/892211203 >> 172.21.15.98:0/892211203 conn(0x7f7bc40fb1d0 msgr2=0x7f7bc40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- 172.21.15.98:0/892211203 shutdown_connections 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- 172.21.15.98:0/892211203 wait complete. 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 Processor -- start 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.680+0000 7f7bca72d700 1 -- start start 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 0x7f7bc41972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc41017f0 0x7f7bc419e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc419e870 con 0x7f7bc40ffc60 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc419e9e0 con 0x7f7bc4100e50 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc419eba0 con 0x7f7bc41017f0 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35150/0 (socket says 172.21.15.98:35150) 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 -- 172.21.15.98:0/4126885327 learned_addr learned my addr 172.21.15.98:0/4126885327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 0x7f7bc41972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc41017f0 0x7f7bc419e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 -- 172.21.15.98:0/4126885327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc41017f0 msgr2=0x7f7bc419e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc41017f0 0x7f7bc419e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 -- 172.21.15.98:0/4126885327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 msgr2=0x7f7bc41972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 0x7f7bc41972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 -- 172.21.15.98:0/4126885327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc009d00 con 0x7f7bc40ffc60 2024-03-18T20:25:42.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400c970 tx=0x7f7bb400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc17fa700 1 -- 172.21.15.98:0/4126885327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4004220 con 0x7f7bc40ffc60 2024-03-18T20:25:42.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bc17fa700 1 -- 172.21.15.98:0/4126885327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7bb4007d30 con 0x7f7bc40ffc60 2024-03-18T20:25:42.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.681+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc419ee80 con 0x7f7bc40ffc60 2024-03-18T20:25:42.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.682+0000 7f7bc17fa700 1 -- 172.21.15.98:0/4126885327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4003b80 con 0x7f7bc40ffc60 2024-03-18T20:25:42.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.682+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc419f2a0 con 0x7f7bc40ffc60 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.684+0000 7f7bc17fa700 1 -- 172.21.15.98:0/4126885327 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f7bb4003ce0 con 0x7f7bc40ffc60 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.684+0000 7f7bc17fa700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 0x7f7bac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.684+0000 7f7bc17fa700 1 -- 172.21.15.98:0/4126885327 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7bb40cc850 con 0x7f7bc40ffc60 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.684+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 0x7f7bac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 0x7f7bac0a3670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8005950 tx=0x7f7bb800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 Processor -- start 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 -- start start 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 0x5632c56d3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c562b940 con 0x5632c56d43d0 2024-03-18T20:25:42.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c5629300 con 0x5632c56d3530 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c5624bf0 con 0x5632c56d2bd0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35156/0 (socket says 172.21.15.98:35156) 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 -- 172.21.15.98:0/3392900192 learned_addr learned my addr 172.21.15.98:0/3392900192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 0x5632c56d3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 -- 172.21.15.98:0/3392900192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 msgr2=0x5632c56d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56d2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 -- 172.21.15.98:0/3392900192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 msgr2=0x5632c56d3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 0x5632c56d3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.685+0000 7f7bc8cca700 1 -- 172.21.15.98:0/3392900192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632c54a6360 con 0x5632c56d43d0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 0x5632c56d3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc015000 tx=0x7f7bbc009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51aa4af5bf96e77a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7baa7fc700 1 -- 172.21.15.98:0/3392900192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00e050 con 0x5632c56d43d0 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7baa7fc700 1 -- 172.21.15.98:0/3392900192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7bbc0076e0 con 0x5632c56d43d0 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7baa7fc700 1 -- 172.21.15.98:0/3392900192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc0052a0 con 0x5632c56d43d0 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3392900192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 msgr2=0x5632c56d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc015000 tx=0x7f7bbc009c80 comp rx=0 tx=0).stop 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3392900192 shutdown_connections 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d43d0 0x5632c56d4850 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56d2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.686+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3392900192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d3530 0x5632c56d3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3392900192 >> 172.21.15.98:0/3392900192 conn(0x5632c56481d0 msgr2=0x5632c56cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3392900192 shutdown_connections 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3392900192 wait complete. 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 -- start start 2024-03-18T20:25:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 Processor -- start 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d2bd0 0x5632c56dca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.687+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d43d0 0x5632c56e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c55c0d30 con 0x5632c56d3530 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c55c09d0 con 0x5632c56d2bd0 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c54a8650 con 0x5632c56d43d0 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 10 client.0 _ll_register_callbacks cb 0x5632c542c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35158/0 (socket says 172.21.15.98:35158) 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 -- 172.21.15.98:0/1518072346 learned_addr learned my addr 172.21.15.98:0/1518072346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d43d0 0x5632c56e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d2bd0 0x5632c56dca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 -- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d43d0 msgr2=0x5632c56e3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d43d0 0x5632c56e3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 -- 172.21.15.98:0/1518072346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d2bd0 msgr2=0x5632c56dca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d2bd0 0x5632c56dca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 -- 172.21.15.98:0/1518072346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc41089f0 con 0x5632c56d3530 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80116c0 tx=0x7f7bb80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becc66493f6ffea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7ba37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb801e830 con 0x5632c56d3530 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7bb8017070 con 0x5632c56d3530 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5632c56c5a80 con 0x5632c56d3530 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bb801e990 con 0x5632c56d3530 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 10 client.15201 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.688+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5632c5452b10 con 0x5632c56d3530 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7bb8032080 con 0x5632c56d3530 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7bb8027a70 con 0x5632c56d3530 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 10 client.15201 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 10 client.15201 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 20 client.15201 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c56e7c00 0x5632c56ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 4 client.15201 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 --> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632c561d6b0 con 0x5632c56e7c00 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.689+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c56e7c00 0x5632c56ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c56e7c00 0x5632c56ecf20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13d951ba71a4656c server_cookie=9e0f706296fb934b in_seq=0 out_seq=0 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7ba37fe700 10 client.15201 ms_handle_connect on v2:172.21.15.169:6838/2458510608 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7ba37fe700 1 -- 172.21.15.98:0/1518072346 <== mds.1 v2:172.21.15.169:6838/2458510608 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5632c561d6b0 con 0x5632c56e7c00 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7ba37fe700 10 client.15201 handle_command_reply: tid=0 2024-03-18T20:25:42.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 client.15201 shutdown 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 msgr2=0x5632c56dcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80116c0 tx=0x7f7bb80047c0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 shutdown_connections 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c56e7c00 0x5632c56ecf20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d43d0 0x5632c56e3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7ba37fe700 0 client.15201 ms_handle_reset on v2:172.21.15.169:6838/2458510608 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d2bd0 0x5632c56dca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/1518072346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56dcfb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.690+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 >> 172.21.15.98:0/1518072346 conn(0x5632c56481d0 msgr2=0x5632c56cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.691+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 shutdown_connections 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.691+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/1518072346 wait complete. 2024-03-18T20:25:42.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.691+0000 7f7bd68c2b80 20 client.15201 trim_cache size 0 max 16384 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 Processor -- start 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 -- start start 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 0x5632c56e3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56da8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c55e2ad0 con 0x5632c56d2bd0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c56988f0 con 0x5632c56d43d0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.708+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c56dd920 con 0x5632c56d3530 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35172/0 (socket says 172.21.15.98:35172) 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 -- 172.21.15.98:0/3395317749 learned_addr learned my addr 172.21.15.98:0/3395317749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56da8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 0x5632c56e3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 -- 172.21.15.98:0/3395317749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 msgr2=0x5632c56e3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 0x5632c56e3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 -- 172.21.15.98:0/3395317749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 msgr2=0x5632c56da8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56da8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 -- 172.21.15.98:0/3395317749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632c55eae50 con 0x5632c56d2bd0 2024-03-18T20:25:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 0x5632c56e3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bb40074a0 tx=0x7f7bb40b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341bb2143c22b02d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7baaffd700 1 -- 172.21.15.98:0/3395317749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb409a050 con 0x5632c56d2bd0 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7baaffd700 1 -- 172.21.15.98:0/3395317749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7bb4099950 con 0x5632c56d2bd0 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.709+0000 7f7baaffd700 1 -- 172.21.15.98:0/3395317749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40ca260 con 0x5632c56d2bd0 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3395317749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 msgr2=0x5632c56d47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7bb40074a0 tx=0x7f7bb40b9040 comp rx=0 tx=0).stop 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3395317749 shutdown_connections 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56da8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d2bd0 0x5632c56d47b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/3395317749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d3530 0x5632c56e3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3395317749 >> 172.21.15.98:0/3395317749 conn(0x5632c56481d0 msgr2=0x5632c56dbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3395317749 shutdown_connections 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/3395317749 wait complete. 2024-03-18T20:25:42.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 -- start start 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.710+0000 7f7bd68c2b80 1 Processor -- start 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56df150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56df780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c573c510 con 0x5632c56d3530 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c573c650 con 0x5632c56d43d0 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c573c7c0 con 0x5632c56d2bd0 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 10 client.0 _ll_register_callbacks cb 0x7f7bb8010d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35180/0 (socket says 172.21.15.98:35180) 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 -- 172.21.15.98:0/2906791186 learned_addr learned my addr 172.21.15.98:0/2906791186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 -- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 msgr2=0x5632c56df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56df150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc8cca700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56df780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 -- 172.21.15.98:0/2906791186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 msgr2=0x5632c56df780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56df780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 -- 172.21.15.98:0/2906791186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc419f2a0 con 0x5632c56d3530 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bc37fe700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8000c00 tx=0x7f7bb800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6865d99d4bcdf1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7ba37fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8004c40 con 0x5632c56d3530 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7bb8017050 con 0x5632c56d3530 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bbc0076e0 con 0x5632c56d3530 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bb802e530 con 0x5632c56d3530 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 10 client.15207 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.711+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5632c54aac80 con 0x5632c56d3530 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7bbc0076e0 con 0x5632c56d3530 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7bb8005160 con 0x5632c56d3530 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 10 client.15207 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 10 client.15207 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 20 client.15207 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c5741610 0x5632c573df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 4 client.15207 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 --> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5632c5473ca0 con 0x5632c5741610 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c5741610 0x5632c573df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7ba37fe700 10 client.15207 ms_handle_connect on v2:172.21.15.169:6838/2458510608 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.712+0000 7f7bc3fff700 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c5741610 0x5632c573df40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd334916af3d5909 server_cookie=9f0471c2bd821a06 in_seq=0 out_seq=0 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7ba37fe700 1 -- 172.21.15.98:0/2906791186 <== mds.1 v2:172.21.15.169:6838/2458510608 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5632c5473ca0 con 0x5632c5741610 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7ba37fe700 10 client.15207 handle_command_reply: tid=0 2024-03-18T20:25:42.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 client.15207 shutdown 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 msgr2=0x5632c56e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8000c00 tx=0x7f7bb800bb20 comp rx=0 tx=0).stop 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 shutdown_connections 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x5632c5741610 0x5632c573df40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c56d43d0 0x5632c56df780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632c56d2bd0 0x5632c56df150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7ba37fe700 0 client.15207 ms_handle_reset on v2:172.21.15.169:6838/2458510608 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 --2- 172.21.15.98:0/2906791186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5632c56d3530 0x5632c56e3fa0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 >> 172.21.15.98:0/2906791186 conn(0x5632c56481d0 msgr2=0x5632c56cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 shutdown_connections 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 1 -- 172.21.15.98:0/2906791186 wait complete. 2024-03-18T20:25:42.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bd68c2b80 20 client.15207 trim_cache size 0 max 16384 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24475, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 44.251539870999999, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:25:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 72.240878581000004 2024-03-18T20:25:42.725 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:25:42.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 msgr2=0x7f7bac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 0x7f7bac0a3670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8005950 tx=0x7f7bb800b410 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 msgr2=0x7f7bc4196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400c970 tx=0x7f7bb400cc80 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 shutdown_connections 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bc41017f0 0x7f7bc419e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bc40ffc60 0x7f7bc4196d80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.713+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f7bac0a11b0 0x7f7bac0a3670 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.714+0000 7f7bca72d700 1 --2- 172.21.15.98:0/4126885327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bc4100e50 0x7f7bc41972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.714+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 >> 172.21.15.98:0/4126885327 conn(0x7f7bc40fb1d0 msgr2=0x7f7bc4067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.714+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 shutdown_connections 2024-03-18T20:25:42.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.714+0000 7f7bca72d700 1 -- 172.21.15.98:0/4126885327 wait complete. 2024-03-18T20:25:42.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T20:25:42.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- 172.21.15.98:0/2109702866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 msgr2=0x7fe6781091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- 172.21.15.98:0/2109702866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe6781091c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe67000b0d0 tx=0x7fe67000b3e0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- 172.21.15.98:0/2109702866 shutdown_connections 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- 172.21.15.98:0/2109702866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe6781091c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- 172.21.15.98:0/2109702866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678100d80 0x7fe6781011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- 172.21.15.98:0/2109702866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678101f70 0x7fe6781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- 172.21.15.98:0/2109702866 >> 172.21.15.98:0/2109702866 conn(0x7fe6780fc2f0 msgr2=0x7fe6780fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- 172.21.15.98:0/2109702866 shutdown_connections 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- 172.21.15.98:0/2109702866 wait complete. 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 Processor -- start 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 -- start start 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678100d80 0x7fe678196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678101f70 0x7fe678197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.849+0000 7fe67f923700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67f923700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819e750 con 0x7fe678102910 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67f923700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819e890 con 0x7fe678101f70 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67f923700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67819ea20 con 0x7fe678100d80 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35190/0 (socket says 172.21.15.98:35190) 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 -- 172.21.15.98:0/3147427592 learned_addr learned my addr 172.21.15.98:0/3147427592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67cebe700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678101f70 0x7fe678197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678100d80 0x7fe678196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 -- 172.21.15.98:0/3147427592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678100d80 msgr2=0x7fe678196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678100d80 0x7fe678196c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 -- 172.21.15.98:0/3147427592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678101f70 msgr2=0x7fe678197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678101f70 0x7fe678197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 -- 172.21.15.98:0/3147427592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009d00 con 0x7fe678102910 2024-03-18T20:25:42.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe670007b60 tx=0x7fe6700094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe66e7fc700 1 -- 172.21.15.98:0/3147427592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000e040 con 0x7fe678102910 2024-03-18T20:25:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe66e7fc700 1 -- 172.21.15.98:0/3147427592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe670004e60 con 0x7fe678102910 2024-03-18T20:25:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe66e7fc700 1 -- 172.21.15.98:0/3147427592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670012a00 con 0x7fe678102910 2024-03-18T20:25:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67819eca0 con 0x7fe678102910 2024-03-18T20:25:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.850+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67819f070 con 0x7fe678102910 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.853+0000 7fe66e7fc700 1 -- 172.21.15.98:0/3147427592 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe6700056e0 con 0x7fe678102910 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.853+0000 7fe66e7fc700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 0x7fe6640a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.854+0000 7fe66e7fc700 1 -- 172.21.15.98:0/3147427592 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe6700ce270 con 0x7fe678102910 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.854+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 0x7fe6640a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.854+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 0x7fe6640a3670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe678101dd0 tx=0x7fe668005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 Processor -- start 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 -- start start 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd0e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd0f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcc62940 con 0x5602bcd0f7e0 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcc60300 con 0x5602bcd0e940 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcc5bbf0 con 0x5602bcd0dfe0 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35194/0 (socket says 172.21.15.98:35194) 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 -- 172.21.15.98:0/3945797531 learned_addr learned my addr 172.21.15.98:0/3945797531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 -- 172.21.15.98:0/3945797531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 msgr2=0x5602bcd0e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd0e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 -- 172.21.15.98:0/3945797531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 msgr2=0x5602bcd0f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd0f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 -- 172.21.15.98:0/3945797531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602bcadd360 con 0x5602bcd0f7e0 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.855+0000 7fe67dec0700 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6700978a0 tx=0x7fe67009af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f507921fb3fa6aba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe6637fe700 1 -- 172.21.15.98:0/3945797531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67009baa0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe6637fe700 1 -- 172.21.15.98:0/3945797531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe6700ce030 con 0x5602bcd0f7e0 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe6637fe700 1 -- 172.21.15.98:0/3945797531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670019350 con 0x5602bcd0f7e0 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 -- 172.21.15.98:0/3945797531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 msgr2=0x5602bcd0fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6700978a0 tx=0x7fe67009af60 comp rx=0 tx=0).stop 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 -- 172.21.15.98:0/3945797531 shutdown_connections 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd0fc60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd0e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/3945797531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd0f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 -- 172.21.15.98:0/3945797531 >> 172.21.15.98:0/3945797531 conn(0x5602bcc7f1d0 msgr2=0x5602bcd04b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 -- 172.21.15.98:0/3945797531 shutdown_connections 2024-03-18T20:25:42.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.856+0000 7fe68bab8b80 1 -- 172.21.15.98:0/3945797531 wait complete. 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 -- start start 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 Processor -- start 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd17ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd181e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcbf7d30 con 0x5602bcd0f7e0 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcbf79d0 con 0x5602bcd0e940 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcadf650 con 0x5602bcd0dfe0 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe68bab8b80 10 client.0 _ll_register_callbacks cb 0x5602bca63350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35202/0 (socket says 172.21.15.98:35202) 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 -- 172.21.15.98:0/893273880 learned_addr learned my addr 172.21.15.98:0/893273880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67cebe700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd181e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 -- 172.21.15.98:0/893273880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 msgr2=0x5602bcd17ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd17ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd17ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 -- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 msgr2=0x5602bcd181e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd181e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.857+0000 7fe67dec0700 1 -- 172.21.15.98:0/893273880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67819f070 con 0x5602bcd0f7e0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe67cebe700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe67dec0700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe6700996f0 tx=0x7fe6700099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce5e36e7dd5a8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67009b3a0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe67009b600 con 0x5602bcd0f7e0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602bccfca80 con 0x5602bcd0f7e0 2024-03-18T20:25:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe67009b7c0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 10 client.15219 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5602bca89b10 con 0x5602bcd0f7e0 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5602bccfca80 con 0x5602bcd0f7e0 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe6700a1160 con 0x5602bcd0f7e0 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 10 client.15219 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 10 client.15219 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 20 client.15219 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd22960 0x5602bcd27c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 4 client.15219 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.858+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602bcc546b0 con 0x5602bcd22960 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.859+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd22960 0x5602bcd27c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.859+0000 7fe660ff9700 10 client.15219 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.859+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd22960 0x5602bcd27c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcaf798355eb3fd7 server_cookie=15d1ce332b2a4d6c in_seq=0 out_seq=0 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe660ff9700 1 -- 172.21.15.98:0/893273880 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5602bcc546b0 con 0x5602bcd22960 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe660ff9700 10 client.15219 handle_command_reply: tid=0 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 client.15219 shutdown 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 msgr2=0x5602bcd1f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe6700996f0 tx=0x7fe6700099e0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 shutdown_connections 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd22960 0x5602bcd27c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1f220 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe660ff9700 0 client.15219 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0dfe0 0x5602bcd17ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/893273880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 >> 172.21.15.98:0/893273880 conn(0x5602bcc7f1d0 msgr2=0x5602bcd041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 shutdown_connections 2024-03-18T20:25:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 1 -- 172.21.15.98:0/893273880 wait complete. 2024-03-18T20:25:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.860+0000 7fe68bab8b80 20 client.15219 trim_cache size 0 max 16384 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 Processor -- start 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 -- start start 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0dfe0 0x5602bcd04a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0e940 0x5602bcd1f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcd18ba0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcc546b0 con 0x5602bcd0dfe0 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcc19ad0 con 0x5602bcd0e940 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35218/0 (socket says 172.21.15.98:35218) 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 -- 172.21.15.98:0/847390191 learned_addr learned my addr 172.21.15.98:0/847390191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67cebe700 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0e940 0x5602bcd1f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 -- 172.21.15.98:0/847390191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0e940 msgr2=0x5602bcd1f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0e940 0x5602bcd1f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 -- 172.21.15.98:0/847390191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0dfe0 msgr2=0x5602bcd04a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0dfe0 0x5602bcd04a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 -- 172.21.15.98:0/847390191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602bcc21e50 con 0x5602bcd0f7e0 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.878+0000 7fe67dec0700 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe6700978a0 tx=0x7fe67009ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56340aa10c7cd48f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe663fff700 1 -- 172.21.15.98:0/847390191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67009b3a0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe663fff700 1 -- 172.21.15.98:0/847390191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe67009f1d0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe663fff700 1 -- 172.21.15.98:0/847390191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67009aaf0 con 0x5602bcd0f7e0 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- 172.21.15.98:0/847390191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 msgr2=0x5602bcd12740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe6700978a0 tx=0x7fe67009ae00 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- 172.21.15.98:0/847390191 shutdown_connections 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0f7e0 0x5602bcd12740 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0dfe0 0x5602bcd04a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/847390191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0e940 0x5602bcd1f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- 172.21.15.98:0/847390191 >> 172.21.15.98:0/847390191 conn(0x5602bcc7f1d0 msgr2=0x5602bcd171b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- 172.21.15.98:0/847390191 shutdown_connections 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- 172.21.15.98:0/847390191 wait complete. 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.879+0000 7fe68bab8b80 1 -- start start 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 Processor -- start 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd1e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcd1f560 con 0x5602bcd0dfe0 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcd77230 con 0x5602bcd0e940 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602bcd77370 con 0x5602bcd0f7e0 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe68bab8b80 10 client.0 _ll_register_callbacks cb 0x7fe67009b190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35234/0 (socket says 172.21.15.98:35234) 2024-03-18T20:25:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 -- 172.21.15.98:0/4071872844 learned_addr learned my addr 172.21.15.98:0/4071872844 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67cebe700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd1e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67dec0700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 -- 172.21.15.98:0/4071872844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0f7e0 msgr2=0x5602bcd1ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 -- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 msgr2=0x5602bcd1e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd1e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 -- 172.21.15.98:0/4071872844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602bca89b10 con 0x5602bcd0dfe0 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe660ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:25:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.880+0000 7fe67d6bf700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5602bcd1f7d0 tx=0x7fe66800ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31f87b6c993595c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668022070 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe6680227a0 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6700ce030 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6680229a0 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 10 client.15225 fetch_fsmap learned FSMap version 23 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5602bcae1c80 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe6700ce030 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe668005020 con 0x5602bcd0dfe0 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 20 client.15225 populate_metadata read hostname 'smithi098' 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd7c1f0 0x5602bcd78b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 4 client.15225 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:25:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5602bcaaaca0 con 0x5602bcd7c1f0 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.881+0000 7fe67cebe700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd7c1f0 0x5602bcd78b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe67cebe700 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd7c1f0 0x5602bcd78b80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3287d19c1bbe09a server_cookie=2285d18bf4e99b88 in_seq=0 out_seq=0 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe660ff9700 10 client.15225 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe660ff9700 1 -- 172.21.15.98:0/4071872844 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5602bcaaaca0 con 0x5602bcd7c1f0 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe660ff9700 10 client.15225 handle_command_reply: tid=0 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 client.15225 shutdown 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 msgr2=0x5602bcd1e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5602bcd1f7d0 tx=0x7fe66800ddf0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 shutdown_connections 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5602bcd7c1f0 0x5602bcd78b80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602bcd0f7e0 0x5602bcd1ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5602bcd0dfe0 0x5602bcd1e3d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 --2- 172.21.15.98:0/4071872844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602bcd0e940 0x5602bcd1e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe660ff9700 0 client.15225 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 >> 172.21.15.98:0/4071872844 conn(0x5602bcc7f1d0 msgr2=0x5602bcd04a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 shutdown_connections 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 1 -- 172.21.15.98:0/4071872844 wait complete. 2024-03-18T20:25:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.882+0000 7fe68bab8b80 20 client.15225 trim_cache size 0 max 16384 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24494, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 61.544387387999997, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:25:42.894 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 67.197502737999997 2024-03-18T20:25:42.895 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:25:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 msgr2=0x7fe6640a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 0x7fe6640a3670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe678101dd0 tx=0x7fe668005d70 comp rx=0 tx=0).stop 2024-03-18T20:25:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 msgr2=0x7fe67819e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe670007b60 tx=0x7fe6700094c0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 shutdown_connections 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe678102910 0x7fe67819e180 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe678100d80 0x7fe678196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fe6640a11b0 0x7fe6640a3670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 --2- 172.21.15.98:0/3147427592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe678101f70 0x7fe678197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 >> 172.21.15.98:0/3147427592 conn(0x7fe6780fc2f0 msgr2=0x7fe678105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 shutdown_connections 2024-03-18T20:25:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:42.883+0000 7fe67f923700 1 -- 172.21.15.98:0/3147427592 wait complete. 2024-03-18T20:25:42.899 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T20:25:42.899 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T20:25:42.909 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:25:42.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T20:25:42.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:42 smithi098 ceph-mon[43462]: pgmap v253: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:42.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:42 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2344906117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:42.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:42 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1922838493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:42 smithi136 ceph-mon[48073]: pgmap v253: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:42 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2344906117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:42.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:42 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1922838493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:42 smithi169 ceph-mon[48529]: pgmap v253: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-18T20:25:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:42 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2344906117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:42 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1922838493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- 172.21.15.98:0/2673038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c0ffcf0 msgr2=0x7fcf3c100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 --2- 172.21.15.98:0/2673038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c100110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0099c0 tx=0x7fcf2c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- 172.21.15.98:0/2673038 shutdown_connections 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 --2- 172.21.15.98:0/2673038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c101890 0x7fcf3c108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 --2- 172.21.15.98:0/2673038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c100110 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 --2- 172.21.15.98:0/2673038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 0x7fcf3c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- 172.21.15.98:0/2673038 >> 172.21.15.98:0/2673038 conn(0x7fcf3c0fb2b0 msgr2=0x7fcf3c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- 172.21.15.98:0/2673038 shutdown_connections 2024-03-18T20:25:44.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- 172.21.15.98:0/2673038 wait complete. 2024-03-18T20:25:44.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 Processor -- start 2024-03-18T20:25:44.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.063+0000 7fcf424de700 1 -- start start 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 0x7fcf3c1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c19d760 con 0x7fcf3c101890 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c19d8a0 con 0x7fcf3c100ef0 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf424de700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c19da30 con 0x7fcf3c0ffcf0 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54684/0 (socket says 172.21.15.98:54684) 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 -- 172.21.15.98:0/3789383165 learned_addr learned my addr 172.21.15.98:0/3789383165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf3bfff700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf3b7fe700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 0x7fcf3c1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 -- 172.21.15.98:0/3789383165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c0ffcf0 msgr2=0x7fcf3c195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 -- 172.21.15.98:0/3789383165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 msgr2=0x7fcf3c1961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 0x7fcf3c1961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.064+0000 7fcf40a7b700 1 -- 172.21.15.98:0/3789383165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c0095f0 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf40a7b700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400b9a0 tx=0x7fcf3400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3400c930 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf34014440 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c19dd10 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf340075d0 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.065+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c19e1c0 con 0x7fcf3c101890 2024-03-18T20:25:44.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.066+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3c107260 con 0x7fcf3c101890 2024-03-18T20:25:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.067+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fcf340145b0 con 0x7fcf3c101890 2024-03-18T20:25:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.068+0000 7fcf397fa700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 0x7fcf240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.068+0000 7fcf3bfff700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 0x7fcf240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.068+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcf34006e40 con 0x7fcf3c101890 2024-03-18T20:25:44.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.068+0000 7fcf3bfff700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 0x7fcf240a3670 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006010 tx=0x7fcf2c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.071+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcf340960d0 con 0x7fcf3c101890 2024-03-18T20:25:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.280+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf3c0611f0 con 0x7fcf240a11b0 2024-03-18T20:25:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.281+0000 7fcf397fa700 1 -- 172.21.15.98:0/3789383165 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcf3c0611f0 con 0x7fcf240a11b0 2024-03-18T20:25:44.284 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-03-18T20:25:44.284 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-03-18T20:25:44.284 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-03-18T20:25:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 msgr2=0x7fcf240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 0x7fcf240a3670 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006010 tx=0x7fcf2c000bc0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 msgr2=0x7fcf3c19d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400b9a0 tx=0x7fcf3400bcb0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 shutdown_connections 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf3c101890 0x7fcf3c19d190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf3c0ffcf0 0x7fcf3c195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fcf240a11b0 0x7fcf240a3670 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 --2- 172.21.15.98:0/3789383165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf3c100ef0 0x7fcf3c1961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 >> 172.21.15.98:0/3789383165 conn(0x7fcf3c0fb2b0 msgr2=0x7fcf3c1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 shutdown_connections 2024-03-18T20:25:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:44.283+0000 7fcf424de700 1 -- 172.21.15.98:0/3789383165 wait complete. 2024-03-18T20:25:44.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T20:25:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:44 smithi136 ceph-mon[48073]: pgmap v254: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-18T20:25:45.056 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:44 smithi098 ceph-mon[43462]: pgmap v254: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-18T20:25:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:44 smithi169 ceph-mon[48529]: pgmap v254: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 -- 172.21.15.98:0/2232967160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df00ffcf0 msgr2=0x7f6df0100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 --2- 172.21.15.98:0/2232967160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df00ffcf0 0x7f6df0100110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6de40099c0 tx=0x7f6de4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 -- 172.21.15.98:0/2232967160 shutdown_connections 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 --2- 172.21.15.98:0/2232967160 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df0101890 0x7f6df0108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 --2- 172.21.15.98:0/2232967160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df00ffcf0 0x7f6df0100110 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 --2- 172.21.15.98:0/2232967160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6df0100ef0 0x7f6df0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 -- 172.21.15.98:0/2232967160 >> 172.21.15.98:0/2232967160 conn(0x7f6df00fb2b0 msgr2=0x7f6df00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 -- 172.21.15.98:0/2232967160 shutdown_connections 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 -- 172.21.15.98:0/2232967160 wait complete. 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.923+0000 7f6df6b61700 1 Processor -- start 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 -- start start 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df00ffcf0 0x7f6df0069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6df0100ef0 0x7f6df019b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df019e730 con 0x7f6df0101890 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df019e8a0 con 0x7f6df0100ef0 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df6b61700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df019ea60 con 0x7f6df00ffcf0 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54694/0 (socket says 172.21.15.98:54694) 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 -- 172.21.15.98:0/1871569952 learned_addr learned my addr 172.21.15.98:0/1871569952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df48fd700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df00ffcf0 0x7f6df0069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 -- 172.21.15.98:0/1871569952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df00ffcf0 msgr2=0x7f6df0069b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df00ffcf0 0x7f6df0069b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 -- 172.21.15.98:0/1871569952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6df0100ef0 msgr2=0x7f6df019b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6df0100ef0 0x7f6df019b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 -- 172.21.15.98:0/1871569952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40095f0 con 0x7f6df0101890 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.924+0000 7f6df50fe700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6de800ada0 tx=0x7f6de800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.925+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de800f070 con 0x7f6df0101890 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.925+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df019ed40 con 0x7f6df0101890 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.925+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6de8007b30 con 0x7f6df0101890 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.925+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8005280 con 0x7f6df0101890 2024-03-18T20:25:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.925+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df019f190 con 0x7f6df0101890 2024-03-18T20:25:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.926+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0107210 con 0x7f6df0101890 2024-03-18T20:25:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.927+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f6de800c980 con 0x7f6df0101890 2024-03-18T20:25:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.927+0000 7f6dedffb700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 0x7f6de00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.927+0000 7f6df48fd700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 0x7f6de00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.927+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6de80cc460 con 0x7f6df0101890 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.928+0000 7f6df48fd700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 0x7f6de00a3670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006010 tx=0x7f6de4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:45.931+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6de8095810 con 0x7f6df0101890 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T20:25:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-18T20:25:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T20:25:46.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-18T20:25:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T20:25:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-18T20:25:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.133+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6df00611f0 con 0x7f6de00a11b0 2024-03-18T20:25:46.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.145+0000 7f6dedffb700 1 -- 172.21.15.98:0/1871569952 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4154 (secure 0 0 0) 0x7f6df00611f0 con 0x7f6de00a11b0 2024-03-18T20:25:46.148 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (72s) 6s ago 72s 15.0M - 18.2.2-914-g49bdc511 015bdba7dcef 68a5657ed565 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi136 running (70s) 6s ago 70s 30.6M - 18.2.2-914-g49bdc511 015bdba7dcef fae059698b04 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi169 running (79s) 6s ago 79s 21.3M - 18.2.2-914-g49bdc511 015bdba7dcef dcdda2bc0b1f 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (84s) 6s ago 84s 17.2M - 18.2.2-914-g49bdc511 015bdba7dcef 5880f643492e 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi136 running (83s) 6s ago 82s 16.7M - 18.2.2-914-g49bdc511 015bdba7dcef 47f395313248 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi169 running (81s) 6s ago 81s 19.1M - 18.2.2-914-g49bdc511 015bdba7dcef 6387423f4382 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (74s) 6s ago 73s 15.0M - 18.2.2-914-g49bdc511 015bdba7dcef 3d72b83cddf3 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi136 running (77s) 6s ago 77s 18.7M - 18.2.2-914-g49bdc511 015bdba7dcef b8b747e34d31 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi169 running (75s) 6s ago 75s 20.2M - 18.2.2-914-g49bdc511 015bdba7dcef d42883216776 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (86s) 6s ago 86s 16.4M - 18.2.2-914-g49bdc511 015bdba7dcef 617961d0f97c 2024-03-18T20:25:46.149 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi136 running (88s) 6s ago 88s 17.6M - 18.2.2-914-g49bdc511 015bdba7dcef 6003bbe0872d 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi169 running (90s) 6s ago 89s 16.4M - 18.2.2-914-g49bdc511 015bdba7dcef 8bed39dce3da 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (6m) 6s ago 6m 529M - 18.2.2-914-g49bdc511 015bdba7dcef d79e528d7c55 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi136 *:8443,8765 running (5m) 6s ago 5m 458M - 18.2.2-914-g49bdc511 015bdba7dcef ce4ceee50fc1 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi169 *:8443,8765 running (5m) 6s ago 5m 459M - 18.2.2-914-g49bdc511 015bdba7dcef aaaa03a83ba2 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (6m) 6s ago 6m 95.6M 2048M 18.2.2-914-g49bdc511 015bdba7dcef dd974b9940e3 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi136 running (5m) 6s ago 5m 65.1M 2048M 18.2.2-914-g49bdc511 015bdba7dcef ebdc6a486168 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi169 running (5m) 6s ago 5m 64.2M 2048M 18.2.2-914-g49bdc511 015bdba7dcef 6a951db64f82 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (4m) 6s ago 4m 70.5M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 531c7ddca362 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi136 running (4m) 6s ago 4m 74.5M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 68cbc28ec592 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi169 running (4m) 6s ago 4m 74.5M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 2550e9b117e7 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (4m) 6s ago 4m 63.7M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 7989b402f279 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi136 running (3m) 6s ago 3m 63.4M 4278M 18.2.2-914-g49bdc511 015bdba7dcef f4f467ba53db 2024-03-18T20:25:46.150 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi169 running (3m) 6s ago 3m 65.8M 4278M 18.2.2-914-g49bdc511 015bdba7dcef de57500b7498 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (3m) 6s ago 3m 62.7M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 52eec44858cd 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi136 running (2m) 6s ago 2m 62.8M 4278M 18.2.2-914-g49bdc511 015bdba7dcef eb8a590dac28 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi169 running (2m) 6s ago 2m 61.0M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 7f84e33c9b95 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (2m) 6s ago 2m 59.6M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 0fe3cd60ec2b 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi136 running (2m) 6s ago 2m 62.0M 4278M 18.2.2-914-g49bdc511 015bdba7dcef fbd7011cd2c0 2024-03-18T20:25:46.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi169 running (104s) 6s ago 103s 53.3M 4278M 18.2.2-914-g49bdc511 015bdba7dcef 76b7c49f8eb3 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 msgr2=0x7f6de00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 0x7f6de00a3670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006010 tx=0x7f6de4000bc0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 msgr2=0x7f6df019df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6de800ada0 tx=0x7f6de800c330 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 shutdown_connections 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df0101890 0x7f6df019df90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6df00ffcf0 0x7f6df0069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f6de00a11b0 0x7f6de00a3670 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 --2- 172.21.15.98:0/1871569952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6df0100ef0 0x7f6df019b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 >> 172.21.15.98:0/1871569952 conn(0x7f6df00fb2b0 msgr2=0x7f6df0104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 shutdown_connections 2024-03-18T20:25:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:46.147+0000 7f6df6b61700 1 -- 172.21.15.98:0/1871569952 wait complete. 2024-03-18T20:25:46.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T20:25:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:46.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: pgmap v255: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-18T20:25:46.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:46 smithi098 ceph-mon[43462]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: pgmap v255: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T20:25:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-18T20:25:46.992 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:46 smithi136 ceph-mon[48073]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T20:25:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: pgmap v255: 129 pgs: 129 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-18T20:25:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:46 smithi169 ceph-mon[48529]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T20:25:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 -- 172.21.15.98:0/2340999380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918101f60 msgr2=0x7f99181023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 --2- 172.21.15.98:0/2340999380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918101f60 0x7f99181023e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9908009990 tx=0x7f9908009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 -- 172.21.15.98:0/2340999380 shutdown_connections 2024-03-18T20:25:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 --2- 172.21.15.98:0/2340999380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918102920 0x7f9918109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 --2- 172.21.15.98:0/2340999380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 --2- 172.21.15.98:0/2340999380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918101f60 0x7f99181023e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 -- 172.21.15.98:0/2340999380 >> 172.21.15.98:0/2340999380 conn(0x7f99180fc3a0 msgr2=0x7f99180fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 -- 172.21.15.98:0/2340999380 shutdown_connections 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.920+0000 7f992009b700 1 -- 172.21.15.98:0/2340999380 wait complete. 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 Processor -- start 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 -- start start 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 0x7f99180731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9918073e80 con 0x7f9918102920 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9918073ff0 con 0x7f9918100e00 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f992009b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99181a0c70 con 0x7f9918101f60 2024-03-18T20:25:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f991e638700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f991e638700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54714/0 (socket says 172.21.15.98:54714) 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.921+0000 7f991e638700 1 -- 172.21.15.98:0/1060933749 learned_addr learned my addr 172.21.15.98:0/1060933749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991d636700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 0x7f99180731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991de37700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 -- 172.21.15.98:0/1060933749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 msgr2=0x7f99180731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 0x7f99180731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 -- 172.21.15.98:0/1060933749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 msgr2=0x7f9918074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 -- 172.21.15.98:0/1060933749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080095f0 con 0x7f9918102920 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991d636700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 0x7f99180731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991de37700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f991e638700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f991000b970 tx=0x7f991000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991000c9b0 con 0x7f9918102920 2024-03-18T20:25:47.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9910014440 con 0x7f9918102920 2024-03-18T20:25:47.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99181a0f00 con 0x7f9918102920 2024-03-18T20:25:47.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99100075d0 con 0x7f9918102920 2024-03-18T20:25:47.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.922+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99181a1320 con 0x7f9918102920 2024-03-18T20:25:47.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.923+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9918108280 con 0x7f9918102920 2024-03-18T20:25:47.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.924+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f99100145b0 con 0x7f9918102920 2024-03-18T20:25:47.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.925+0000 7f990effd700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 0x7f99040a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.925+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9910006e40 con 0x7f9918102920 2024-03-18T20:25:47.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.925+0000 7f991de37700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 0x7f99040a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.925+0000 7f991de37700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 0x7f99040a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9914005fd0 tx=0x7f9914005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:47.928+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9910096050 con 0x7f9918102920 2024-03-18T20:25:47.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:47 smithi136 ceph-mon[48073]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:47.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:47 smithi136 ceph-mon[48073]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T20:25:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:47 smithi098 ceph-mon[43462]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:48.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:47 smithi098 ceph-mon[43462]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T20:25:48.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.129+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f99180611f0 con 0x7f99040a11b0 2024-03-18T20:25:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:47 smithi169 ceph-mon[48529]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:47 smithi169 ceph-mon[48529]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T20:25:48.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.135+0000 7f990effd700 1 -- 172.21.15.98:0/1060933749 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f99180611f0 con 0x7f99040a11b0 2024-03-18T20:25:48.138 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T20:25:48.138 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 8s ago 93s smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi169=c;smithi169=f;smithi169=i;smithi169=l;count:12 2024-03-18T20:25:48.139 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 8s ago 5m smithi098=x;smithi136=y;smithi169=z;count:3 2024-03-18T20:25:48.139 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 8s ago 5m smithi098:172.21.15.98=a;smithi136:172.21.15.136=b;smithi169:172.21.15.169=c;count:3 2024-03-18T20:25:48.139 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 8s ago - 2024-03-18T20:25:48.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 msgr2=0x7f99040a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 0x7f99040a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9914005fd0 tx=0x7f9914005e30 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 msgr2=0x7f99180736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f991000b970 tx=0x7f991000bd30 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 shutdown_connections 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9918102920 0x7f99180736e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f99040a11b0 0x7f99040a3670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9918100e00 0x7f9918074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 --2- 172.21.15.98:0/1060933749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9918101f60 0x7f99180731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 >> 172.21.15.98:0/1060933749 conn(0x7f99180fc3a0 msgr2=0x7f99181053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 shutdown_connections 2024-03-18T20:25:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:48.137+0000 7f992009b700 1 -- 172.21.15.98:0/1060933749 wait complete. 2024-03-18T20:25:48.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T20:25:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:48 smithi136 ceph-mon[48073]: pgmap v258: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:25:49.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:48 smithi098 ceph-mon[43462]: pgmap v258: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:25:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:48 smithi169 ceph-mon[48529]: pgmap v258: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:25:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 -- 172.21.15.98:0/4275788329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60101880 msgr2=0x7fbb60103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/4275788329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60101880 0x7fbb60103c70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009990 tx=0x7fbb50009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 -- 172.21.15.98:0/4275788329 shutdown_connections 2024-03-18T20:25:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/4275788329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb60106ae0 0x7fbb60108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/4275788329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60101880 0x7fbb60103c70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/4275788329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb601041b0 0x7fbb601065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 -- 172.21.15.98:0/4275788329 >> 172.21.15.98:0/4275788329 conn(0x7fbb600fb2b0 msgr2=0x7fbb600fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 -- 172.21.15.98:0/4275788329 shutdown_connections 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.722+0000 7fbb66a0b700 1 -- 172.21.15.98:0/4275788329 wait complete. 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 Processor -- start 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 -- start start 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 0x7fbb60101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb601041b0 0x7fbb600ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb60100510 con 0x7fbb60106ae0 2024-03-18T20:25:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb60100680 con 0x7fbb60101880 2024-03-18T20:25:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb66a0b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb601a0c60 con 0x7fbb601041b0 2024-03-18T20:25:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb64fa8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb64fa8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54742/0 (socket says 172.21.15.98:54742) 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb64fa8700 1 -- 172.21.15.98:0/3613292600 learned_addr learned my addr 172.21.15.98:0/3613292600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb5f7fe700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb601041b0 0x7fbb600ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.723+0000 7fbb5ffff700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 0x7fbb60101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 -- 172.21.15.98:0/3613292600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb601041b0 msgr2=0x7fbb600ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb601041b0 0x7fbb600ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 -- 172.21.15.98:0/3613292600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 msgr2=0x7fbb60101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 0x7fbb60101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 -- 172.21.15.98:0/3613292600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500095f0 con 0x7fbb60106ae0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb5ffff700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 0x7fbb60101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb64fa8700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbb58007d40 tx=0x7fbb5800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800ad40 con 0x7fbb60106ae0 2024-03-18T20:25:49.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbb58007e90 con 0x7fbb60106ae0 2024-03-18T20:25:49.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb601a0ef0 con 0x7fbb60106ae0 2024-03-18T20:25:49.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb580115e0 con 0x7fbb60106ae0 2024-03-18T20:25:49.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.724+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb601a1310 con 0x7fbb60106ae0 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.725+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb600fce30 con 0x7fbb60106ae0 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.726+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbb58012420 con 0x7fbb60106ae0 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.727+0000 7fbb5d7fa700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 0x7fbb480a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.727+0000 7fbb5ffff700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 0x7fbb480a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.727+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbb580cce80 con 0x7fbb60106ae0 2024-03-18T20:25:49.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.727+0000 7fbb5ffff700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 0x7fbb480a3670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbb50003780 tx=0x7fbb50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:49.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.731+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbb580962b0 con 0x7fbb60106ae0 2024-03-18T20:25:49.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.931+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb600611f0 con 0x7fbb480a11b0 2024-03-18T20:25:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.933+0000 7fbb5d7fa700 1 -- 172.21.15.98:0/3613292600 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fbb600611f0 con 0x7fbb480a11b0 2024-03-18T20:25:49.936 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-03-18T20:25:49.936 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-03-18T20:25:49.937 INFO:teuthology.orchestra.run.smithi098.stdout:smithi136 172.21.15.136 2024-03-18T20:25:49.937 INFO:teuthology.orchestra.run.smithi098.stdout:smithi169 172.21.15.169 2024-03-18T20:25:49.937 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-03-18T20:25:49.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 msgr2=0x7fbb480a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 0x7fbb480a3670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbb50003780 tx=0x7fbb50005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 msgr2=0x7fbb600ffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbb58007d40 tx=0x7fbb5800a340 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 shutdown_connections 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb60106ae0 0x7fbb600ffd70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbb480a11b0 0x7fbb480a3670 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb60101880 0x7fbb60101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 --2- 172.21.15.98:0/3613292600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbb601041b0 0x7fbb600ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 >> 172.21.15.98:0/3613292600 conn(0x7fbb600fb2b0 msgr2=0x7fbb601022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 shutdown_connections 2024-03-18T20:25:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:49.935+0000 7fbb66a0b700 1 -- 172.21.15.98:0/3613292600 wait complete. 2024-03-18T20:25:49.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:49 smithi136 ceph-mon[48073]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:50.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:49 smithi098 ceph-mon[43462]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:50.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:49 smithi169 ceph-mon[48529]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:50.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T20:25:50.543 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:50.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:50 smithi136 ceph-mon[48073]: pgmap v259: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:25:51.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:50 smithi098 ceph-mon[43462]: pgmap v259: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:25:51.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:50 smithi169 ceph-mon[48529]: pgmap v259: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 -- 172.21.15.98:0/1314403263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102a30 msgr2=0x7f4a8c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 --2- 172.21.15.98:0/1314403263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102a30 0x7f4a8c1092e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400b0a0 tx=0x7f4a8400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 -- 172.21.15.98:0/1314403263 shutdown_connections 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 --2- 172.21.15.98:0/1314403263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102a30 0x7f4a8c1092e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 --2- 172.21.15.98:0/1314403263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 0x7f4a8c0ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 --2- 172.21.15.98:0/1314403263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102090 0x7f4a8c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 -- 172.21.15.98:0/1314403263 >> 172.21.15.98:0/1314403263 conn(0x7f4a8c0faa10 msgr2=0x7f4a8c0fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 -- 172.21.15.98:0/1314403263 shutdown_connections 2024-03-18T20:25:51.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.512+0000 7f4a9105a700 1 -- 172.21.15.98:0/1314403263 wait complete. 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 Processor -- start 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 -- start start 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 0x7f4a8c1019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 0x7f4a8c100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c100d20 con 0x7f4a8c102090 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c100e90 con 0x7f4a8c102a30 2024-03-18T20:25:51.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a9105a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c1a0e60 con 0x7f4a8c0ff0b0 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54760/0 (socket says 172.21.15.98:54760) 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a8a59c700 1 -- 172.21.15.98:0/976768961 learned_addr learned my addr 172.21.15.98:0/976768961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.513+0000 7f4a8ad9d700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 0x7f4a8c1019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8b59e700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 0x7f4a8c100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 -- 172.21.15.98:0/976768961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 msgr2=0x7f4a8c1019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 0x7f4a8c1019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 -- 172.21.15.98:0/976768961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 msgr2=0x7f4a8c100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 0x7f4a8c100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 -- 172.21.15.98:0/976768961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84009d00 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8b59e700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 0x7f4a8c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a8a59c700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00b530 tx=0x7f4a7c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c011830 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c1a10f0 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a7c011e20 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00f2e0 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.514+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c1a1670 con 0x7f4a8c102090 2024-03-18T20:25:51.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.515+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8c108390 con 0x7f4a8c102090 2024-03-18T20:25:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.516+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4a7c011990 con 0x7f4a8c102090 2024-03-18T20:25:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.516+0000 7f4a83fff700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 0x7f4a780a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.517+0000 7f4a8ad9d700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 0x7f4a780a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.517+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4a7c0ccbe0 con 0x7f4a8c102090 2024-03-18T20:25:51.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.517+0000 7f4a8ad9d700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 0x7f4a780a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c068550 tx=0x7f4a74009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:51.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.520+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a7c095fb0 con 0x7f4a8c102090 2024-03-18T20:25:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.722+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a8c0611f0 con 0x7f4a780a11b0 2024-03-18T20:25:51.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.723+0000 7f4a83fff700 1 -- 172.21.15.98:0/976768961 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f4a8c0611f0 con 0x7f4a780a11b0 2024-03-18T20:25:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T20:25:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T20:25:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:smithi136 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD514100122P0SGN 1863G No 2m ago Has a FileSystem, LVM detected 2024-03-18T20:25:51.727 INFO:teuthology.orchestra.run.smithi098.stdout:smithi169 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300LN400BGN 372G No 104s ago Has a FileSystem, LVM detected 2024-03-18T20:25:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 msgr2=0x7f4a780a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 0x7f4a780a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c068550 tx=0x7f4a74009190 comp rx=0 tx=0).stop 2024-03-18T20:25:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 msgr2=0x7f4a8c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00b530 tx=0x7f4a7c00b840 comp rx=0 tx=0).stop 2024-03-18T20:25:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 shutdown_connections 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4a780a11b0 0x7f4a780a3670 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8c102a30 0x7f4a8c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a8c0ff0b0 0x7f4a8c1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 --2- 172.21.15.98:0/976768961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c102090 0x7f4a8c100040 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.725+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 >> 172.21.15.98:0/976768961 conn(0x7f4a8c0faa10 msgr2=0x7f4a8c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.726+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 shutdown_connections 2024-03-18T20:25:51.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:51.726+0000 7f4a9105a700 1 -- 172.21.15.98:0/976768961 wait complete. 2024-03-18T20:25:51.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:51 smithi136 ceph-mon[48073]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:52.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:51 smithi098 ceph-mon[43462]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:52.101 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T20:25:52.111 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:25:52.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T20:25:52.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:51 smithi169 ceph-mon[48529]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:52.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:52 smithi136 ceph-mon[48073]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:52.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:52 smithi136 ceph-mon[48073]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:53.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:52 smithi098 ceph-mon[43462]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:53.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:52 smithi098 ceph-mon[43462]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:53.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:52 smithi169 ceph-mon[48529]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:53.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:52 smithi169 ceph-mon[48529]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 -- 172.21.15.98:0/2035177741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 msgr2=0x7f2a20103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 --2- 172.21.15.98:0/2035177741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20103c70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009990 tx=0x7f2a14009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 -- 172.21.15.98:0/2035177741 shutdown_connections 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 --2- 172.21.15.98:0/2035177741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a20106ae0 0x7f2a20108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 --2- 172.21.15.98:0/2035177741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20103c70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 --2- 172.21.15.98:0/2035177741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a201041b0 0x7f2a201065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 -- 172.21.15.98:0/2035177741 >> 172.21.15.98:0/2035177741 conn(0x7f2a200fb2b0 msgr2=0x7f2a200fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 -- 172.21.15.98:0/2035177741 shutdown_connections 2024-03-18T20:25:53.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.311+0000 7f2a24922700 1 -- 172.21.15.98:0/2035177741 wait complete. 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 Processor -- start 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 -- start start 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a201041b0 0x7f2a200ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 0x7f2a200ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:53.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20100510 con 0x7f2a20101880 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20100680 con 0x7f2a20106ae0 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a24922700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a201a0cf0 con 0x7f2a201041b0 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a1e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a1e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41230/0 (socket says 172.21.15.98:41230) 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a1e59c700 1 -- 172.21.15.98:0/860725076 learned_addr learned my addr 172.21.15.98:0/860725076 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.312+0000 7f2a1dd9b700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a201041b0 0x7f2a200ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1ed9d700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 0x7f2a200ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 -- 172.21.15.98:0/860725076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a201041b0 msgr2=0x7f2a200ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a201041b0 0x7f2a200ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 -- 172.21.15.98:0/860725076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 msgr2=0x7f2a200ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 0x7f2a200ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 -- 172.21.15.98:0/860725076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a140095f0 con 0x7f2a20101880 2024-03-18T20:25:53.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1ed9d700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 0x7f2a200ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a1e59c700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098e0 tx=0x7f2a14003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1401d070 con 0x7f2a20101880 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a201a0f20 con 0x7f2a20101880 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2a1400bd90 con 0x7f2a20101880 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14022d20 con 0x7f2a20101880 2024-03-18T20:25:53.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.313+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a201a1280 con 0x7f2a20101880 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.315+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a200fce30 con 0x7f2a20101880 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.315+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f2a14003f40 con 0x7f2a20101880 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.316+0000 7f2a0f7fe700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 0x7f2a080a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.316+0000 7f2a1dd9b700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 0x7f2a080a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.316+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2a140ce360 con 0x7f2a20101880 2024-03-18T20:25:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.316+0000 7f2a1dd9b700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 0x7f2a080a3670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a10005950 tx=0x7f2a1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:53.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.320+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2a140976b0 con 0x7f2a20101880 2024-03-18T20:25:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.560+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a2002d140 con 0x7f2a20101880 2024-03-18T20:25:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.561+0000 7f2a0f7fe700 1 -- 172.21.15.98:0/860725076 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2903 (secure 0 0 0) 0x7f2a14027b70 con 0x7f2a20101880 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:e23 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 23 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-18T20:24:41.335739+0000 2024-03-18T20:25:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-03-18T20:25:00.478029+0000 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 3 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2 2024-03-18T20:25:53.566 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24494,1=24475,2=24461} 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3,4] 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24494} state up:active seq 3 addr [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{0:24454} state up:standby-replay seq 1 addr [v2:172.21.15.169:6836/1340440313,v1:172.21.15.169:6837/1340440313] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{1:24475} state up:active seq 8 addr [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{1:24442} state up:standby-replay seq 1 addr [v2:172.21.15.169:6834/477760816,v1:172.21.15.169:6835/477760816] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{2:24461} state up:active seq 9 addr [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.136:6834/4136937912,v1:172.21.15.136:6835/4136937912] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.98:6834/2703259381,v1:172.21.15.98:6835/2703259381] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.98:6836/1932954082,v1:172.21.15.98:6837/1932954082] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{-1:14592} state up:standby seq 1 addr [v2:172.21.15.98:6838/655758721,v1:172.21.15.98:6839/655758721] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{-1:14604} state up:standby seq 1 addr [v2:172.21.15.98:6840/2638633847,v1:172.21.15.98:6841/2638633847] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{-1:24421} state up:standby seq 1 addr [v2:172.21.15.169:6832/1320002530,v1:172.21.15.169:6833/1320002530] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{-1:24422} state up:standby seq 1 addr [v2:172.21.15.136:6832/1211632092,v1:172.21.15.136:6833/1211632092] compat {c=[1],r=[1],i=[7ff]}] 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 msgr2=0x7f2a080a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 0x7f2a080a3670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a10005950 tx=0x7f2a1000b410 comp rx=0 tx=0).stop 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 msgr2=0x7f2a20101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098e0 tx=0x7f2a14003a50 comp rx=0 tx=0).stop 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 shutdown_connections 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f2a080a11b0 0x7f2a080a3670 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a20106ae0 0x7f2a200ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a20101880 0x7f2a20101190 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 --2- 172.21.15.98:0/860725076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a201041b0 0x7f2a200ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 >> 172.21.15.98:0/860725076 conn(0x7f2a200fb2b0 msgr2=0x7f2a20102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 shutdown_connections 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:53.563+0000 7f2a24922700 1 -- 172.21.15.98:0/860725076 wait complete. 2024-03-18T20:25:53.570 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:53.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:53 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/860725076' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T20:25:53.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T20:25:53.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:53 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/860725076' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T20:25:54.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:53 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/860725076' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:25:54.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:54 smithi098 ceph-mon[43462]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:54.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:54 smithi136 ceph-mon[48073]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 -- 172.21.15.98:0/1019730060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32500ffcf0 msgr2=0x7f3250100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 --2- 172.21.15.98:0/1019730060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32500ffcf0 0x7f3250100110 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f32400099c0 tx=0x7f3240009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 -- 172.21.15.98:0/1019730060 shutdown_connections 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 --2- 172.21.15.98:0/1019730060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3250101890 0x7f3250108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 --2- 172.21.15.98:0/1019730060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32500ffcf0 0x7f3250100110 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 --2- 172.21.15.98:0/1019730060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250100ef0 0x7f3250101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.127+0000 7f325695d700 1 -- 172.21.15.98:0/1019730060 >> 172.21.15.98:0/1019730060 conn(0x7f32500fb2b0 msgr2=0x7f32500fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- 172.21.15.98:0/1019730060 shutdown_connections 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- 172.21.15.98:0/1019730060 wait complete. 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 Processor -- start 2024-03-18T20:25:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- start start 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32500ffcf0 0x7f3250196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 0x7f325019e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325019e890 con 0x7f3250100ef0 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325019e9d0 con 0x7f3250101890 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.128+0000 7f325695d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325019eb90 con 0x7f32500ffcf0 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41240/0 (socket says 172.21.15.98:41240) 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 -- 172.21.15.98:0/2634107795 learned_addr learned my addr 172.21.15.98:0/2634107795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f3254efa700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 0x7f325019e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324ffff700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32500ffcf0 0x7f3250196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 -- 172.21.15.98:0/2634107795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32500ffcf0 msgr2=0x7f3250196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32500ffcf0 0x7f3250196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 -- 172.21.15.98:0/2634107795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 msgr2=0x7f325019e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 0x7f325019e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 -- 172.21.15.98:0/2634107795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400095f0 con 0x7f3250100ef0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f3254efa700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 0x7f325019e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324f7fe700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f324400b530 tx=0x7f324400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244011830 con 0x7f3250100ef0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3244011e20 con 0x7f3250100ef0 2024-03-18T20:25:55.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325019ee70 con 0x7f3250100ef0 2024-03-18T20:25:55.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244010dd0 con 0x7f3250100ef0 2024-03-18T20:25:55.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.129+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325019f2e0 con 0x7f3250100ef0 2024-03-18T20:25:55.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.130+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325004ea80 con 0x7f3250100ef0 2024-03-18T20:25:55.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:54 smithi169 ceph-mon[48529]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 op/s; 88 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.131+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f3244011990 con 0x7f3250100ef0 2024-03-18T20:25:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.132+0000 7f324d7fa700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 0x7f32380a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.132+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f32440ccb40 con 0x7f3250100ef0 2024-03-18T20:25:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.132+0000 7f324ffff700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 0x7f32380a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.132+0000 7f324ffff700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 0x7f32380a36f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3240009990 tx=0x7f3240005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.136+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3244095f70 con 0x7f3250100ef0 2024-03-18T20:25:55.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.327+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f325019f6d0 con 0x7f3250100ef0 2024-03-18T20:25:55.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.328+0000 7f324d7fa700 1 -- 172.21.15.98:0/2634107795 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5602 (secure 0 0 0) 0x7f3244099590 con 0x7f3250100ef0 2024-03-18T20:25:55.331 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 87 2024-03-18T20:25:55.331 INFO:teuthology.orchestra.run.smithi098.stdout:fsid a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:25:55.331 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-18T20:18:57.763866+0000 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-03-18T20:25:46.679841+0000 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 27 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release reef 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T20:25:55.332 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/363521185,v1:172.21.15.98:6803/363521185] [v2:172.21.15.98:6804/363521185,v1:172.21.15.98:6805/363521185] exists,up 9d91034c-3502-4016-9090-515df19fa756 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/1522927033,v1:172.21.15.136:6801/1522927033] [v2:172.21.15.136:6802/1522927033,v1:172.21.15.136:6803/1522927033] exists,up cf985f1e-4bad-4e71-9cf6-abea91ae0378 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6800/151861743,v1:172.21.15.169:6801/151861743] [v2:172.21.15.169:6802/151861743,v1:172.21.15.169:6803/151861743] exists,up 14a75dc5-6809-4c9f-9222-8e6a71c62a82 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/2727695950,v1:172.21.15.98:6811/2727695950] [v2:172.21.15.98:6812/2727695950,v1:172.21.15.98:6813/2727695950] exists,up dc4d0794-a030-42e7-8cdd-6e2c636e56ea 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 32 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/4061852500,v1:172.21.15.136:6809/4061852500] [v2:172.21.15.136:6810/4061852500,v1:172.21.15.136:6811/4061852500] exists,up 63fde25b-a451-4031-8795-9a05f0f2a29d 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 37 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6808/123568177,v1:172.21.15.169:6809/123568177] [v2:172.21.15.169:6810/123568177,v1:172.21.15.169:6811/123568177] exists,up 62b9ea73-b07c-4aa1-8d5d-57fdf0204185 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 44 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/2163470647,v1:172.21.15.98:6819/2163470647] [v2:172.21.15.98:6820/2163470647,v1:172.21.15.98:6821/2163470647] exists,up 2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 49 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/1282488647,v1:172.21.15.136:6817/1282488647] [v2:172.21.15.136:6818/1282488647,v1:172.21.15.136:6819/1282488647] exists,up df65208c-eee4-455f-b0e0-9a42f261531e 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 56 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6816/3298544672,v1:172.21.15.169:6817/3298544672] [v2:172.21.15.169:6818/3298544672,v1:172.21.15.169:6819/3298544672] exists,up 5ff34b6f-b1af-4751-afda-9fa4bc03e881 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 63 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/1957300736,v1:172.21.15.98:6827/1957300736] [v2:172.21.15.98:6828/1957300736,v1:172.21.15.98:6829/1957300736] exists,up 5969a1df-7157-4720-bb4e-d499f9a2d3d3 2024-03-18T20:25:55.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 68 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/3492008064,v1:172.21.15.136:6825/3492008064] [v2:172.21.15.136:6826/3492008064,v1:172.21.15.136:6827/3492008064] exists,up dbb5b70e-d702-48c4-9811-d1d75e09de82 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 76 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6824/2513253907,v1:172.21.15.169:6825/2513253907] [v2:172.21.15.169:6826/2513253907,v1:172.21.15.169:6827/2513253907] exists,up 50289373-3457-45bd-8c72-de8ae15b6ec2 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.5 [8,11] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.b [2,6] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.10 [2,4] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1a [8,1] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1b [5,11] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1d [2,4] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.11 [4,7] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.27 [4,8] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.2c [4,9] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.39 [5,6] 2024-03-18T20:25:55.334 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1287611061 expires 2024-03-19T20:19:45.512552+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1287611061 expires 2024-03-19T20:19:45.512552+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2681086384 expires 2024-03-19T20:19:25.585401+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3671856112 expires 2024-03-19T20:19:45.512552+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1417871753 expires 2024-03-19T20:19:25.585401+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2768304991 expires 2024-03-19T20:19:45.512552+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/479479747 expires 2024-03-19T20:19:25.585401+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/479479747 expires 2024-03-19T20:19:25.585401+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2082705134 expires 2024-03-19T20:19:45.512552+0000 2024-03-18T20:25:55.335 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3238391768 expires 2024-03-19T20:19:25.585401+0000 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 msgr2=0x7f32380a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 0x7f32380a36f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3240009990 tx=0x7f3240005ec0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 msgr2=0x7f32501972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f324400b530 tx=0x7f324400b8f0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 shutdown_connections 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f32380a1230 0x7f32380a36f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3250101890 0x7f325019e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32500ffcf0 0x7f3250196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 --2- 172.21.15.98:0/2634107795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3250100ef0 0x7f32501972d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 >> 172.21.15.98:0/2634107795 conn(0x7f32500fb2b0 msgr2=0x7f3250068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 shutdown_connections 2024-03-18T20:25:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.330+0000 7f325695d700 1 -- 172.21.15.98:0/2634107795 wait complete. 2024-03-18T20:25:55.717 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T20:25:55.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 -- 172.21.15.98:0/1580417923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530100d80 msgr2=0x7f45301011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 --2- 172.21.15.98:0/1580417923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530100d80 0x7f45301011e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f451c0099c0 tx=0x7f451c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 -- 172.21.15.98:0/1580417923 shutdown_connections 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 --2- 172.21.15.98:0/1580417923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4530101720 0x7f4530107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 --2- 172.21.15.98:0/1580417923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45300ffb90 0x7f45300fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 --2- 172.21.15.98:0/1580417923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530100d80 0x7f45301011e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 -- 172.21.15.98:0/1580417923 >> 172.21.15.98:0/1580417923 conn(0x7f45300fb100 msgr2=0x7f45300fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 -- 172.21.15.98:0/1580417923 shutdown_connections 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.853+0000 7f4536c89700 1 -- 172.21.15.98:0/1580417923 wait complete. 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 Processor -- start 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 -- start start 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45300ffb90 0x7f4530196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4530100d80 0x7f4530197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453019e630 con 0x7f4530101720 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453019e770 con 0x7f45300ffb90 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4536c89700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453019e900 con 0x7f4530100d80 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41256/0 (socket says 172.21.15.98:41256) 2024-03-18T20:25:55.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 -- 172.21.15.98:0/3483181886 learned_addr learned my addr 172.21.15.98:0/3483181886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4534a25700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45300ffb90 0x7f4530196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f452ffff700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4530100d80 0x7f4530197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 -- 172.21.15.98:0/3483181886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4530100d80 msgr2=0x7f4530197040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4530100d80 0x7f4530197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 -- 172.21.15.98:0/3483181886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45300ffb90 msgr2=0x7f4530196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45300ffb90 0x7f4530196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.854+0000 7f4535226700 1 -- 172.21.15.98:0/3483181886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0095f0 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f4535226700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4528007ee0 tx=0x7f452800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4528017070 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453019ebe0 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f452800f040 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4528005710 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.855+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453019f0f0 con 0x7f4530101720 2024-03-18T20:25:55.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.856+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530067180 con 0x7f4530101720 2024-03-18T20:25:55.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.857+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4528005870 con 0x7f4530101720 2024-03-18T20:25:55.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.857+0000 7f452dffb700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 0x7f45200a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.857+0000 7f4534a25700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 0x7f45200a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.857+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45280cc630 con 0x7f4530101720 2024-03-18T20:25:55.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.858+0000 7f4534a25700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 0x7f45200a3670 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4524005fd0 tx=0x7f4524005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:55.861+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4528095950 con 0x7f4530101720 2024-03-18T20:25:55.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:55 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2634107795' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T20:25:56.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.068+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f453019f3e0 con 0x7f4530101720 2024-03-18T20:25:56.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.069+0000 7f452dffb700 1 -- 172.21.15.98:0/3483181886 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24488 (secure 0 0 0) 0x7f4528082e60 con 0x7f4530101720 2024-03-18T20:25:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:56.073 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":87,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:25:46.679841+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 msgr2=0x7f45200a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 0x7f45200a3670 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4524005fd0 tx=0x7f4524005dd0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 msgr2=0x7f453019e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4528007ee0 tx=0x7f452800d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 shutdown_connections 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4530101720 0x7f453019e060 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45200a11b0 0x7f45200a3670 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45300ffb90 0x7f4530196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 --2- 172.21.15.98:0/3483181886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4530100d80 0x7f4530197040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 >> 172.21.15.98:0/3483181886 conn(0x7f45300fb100 msgr2=0x7f4530069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 shutdown_connections 2024-03-18T20:25:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.072+0000 7f4536c89700 1 -- 172.21.15.98:0/3483181886 wait complete. 2024-03-18T20:25:56.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:55 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2634107795' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T20:25:56.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:25:56.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:56.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:55 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2634107795' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T20:25:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 -- 172.21.15.98:0/2417896532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 msgr2=0x7fa0e8108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/2417896532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e8108eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000b0a0 tx=0x7fa0e000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 -- 172.21.15.98:0/2417896532 shutdown_connections 2024-03-18T20:25:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/2417896532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e8108eb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/2417896532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e81017e0 0x7fa0e8103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/2417896532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e8104110 0x7fa0e8106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.208+0000 7fa0ed22d700 1 -- 172.21.15.98:0/2417896532 >> 172.21.15.98:0/2417896532 conn(0x7fa0e80fb1d0 msgr2=0x7fa0e80fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.209+0000 7fa0ed22d700 1 -- 172.21.15.98:0/2417896532 shutdown_connections 2024-03-18T20:25:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.209+0000 7fa0ed22d700 1 -- 172.21.15.98:0/2417896532 wait complete. 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.209+0000 7fa0ed22d700 1 Processor -- start 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 -- start start 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 0x7fa0e8196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e8104110 0x7fa0e8197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e819e840 con 0x7fa0e8106a40 2024-03-18T20:25:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e819e9b0 con 0x7fa0e81017e0 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0ed22d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e819eb70 con 0x7fa0e8104110 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41266/0 (socket says 172.21.15.98:41266) 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 -- 172.21.15.98:0/3403552901 learned_addr learned my addr 172.21.15.98:0/3403552901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e659c700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e8104110 0x7fa0e8197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e6d9d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 0x7fa0e8196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 -- 172.21.15.98:0/3403552901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e8104110 msgr2=0x7fa0e8197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e8104110 0x7fa0e8197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 -- 172.21.15.98:0/3403552901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 msgr2=0x7fa0e8196d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 0x7fa0e8196d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e759e700 1 -- 172.21.15.98:0/3403552901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e0009d00 con 0x7fa0e8106a40 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.210+0000 7fa0e6d9d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 0x7fa0e8196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0e759e700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0007a30 tx=0x7fa0e0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000e040 con 0x7fa0e8106a40 2024-03-18T20:25:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e819edf0 con 0x7fa0e8106a40 2024-03-18T20:25:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa0e00056e0 con 0x7fa0e8106a40 2024-03-18T20:25:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e001dad0 con 0x7fa0e8106a40 2024-03-18T20:25:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.211+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e819f1c0 con 0x7fa0e8106a40 2024-03-18T20:25:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.212+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e80fcd50 con 0x7fa0e8106a40 2024-03-18T20:25:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.213+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa0e0005850 con 0x7fa0e8106a40 2024-03-18T20:25:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.213+0000 7fa0cffff700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 0x7fa0d00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.214+0000 7fa0e6d9d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 0x7fa0d00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.214+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa0e00cef20 con 0x7fa0e8106a40 2024-03-18T20:25:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.214+0000 7fa0e6d9d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 0x7fa0d00a3670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc005950 tx=0x7fa0dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.217+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa0e0098270 con 0x7fa0e8106a40 2024-03-18T20:25:56.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.413+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0e802d0d0 con 0x7fa0e8106a40 2024-03-18T20:25:56.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.414+0000 7fa0cffff700 1 -- 172.21.15.98:0/3403552901 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0e009b890 con 0x7fa0e8106a40 2024-03-18T20:25:56.417 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:25:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 msgr2=0x7fa0d00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 0x7fa0d00a3670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc005950 tx=0x7fa0dc0058e0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 msgr2=0x7fa0e819e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0007a30 tx=0x7fa0e0004c80 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 shutdown_connections 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0e8106a40 0x7fa0e819e270 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa0d00a11b0 0x7fa0d00a3670 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0e81017e0 0x7fa0e8196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 --2- 172.21.15.98:0/3403552901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0e8104110 0x7fa0e8197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 >> 172.21.15.98:0/3403552901 conn(0x7fa0e80fb1d0 msgr2=0x7fa0e8102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 shutdown_connections 2024-03-18T20:25:56.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.416+0000 7fa0ed22d700 1 -- 172.21.15.98:0/3403552901 wait complete. 2024-03-18T20:25:56.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.553+0000 7f45eeef3700 1 -- 172.21.15.98:0/1705885957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8102990 msgr2=0x7f45e8109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.553+0000 7f45eeef3700 1 --2- 172.21.15.98:0/1705885957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8102990 0x7f45e8109240 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f45e000b0a0 tx=0x7f45e000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- 172.21.15.98:0/1705885957 shutdown_connections 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- 172.21.15.98:0/1705885957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8102990 0x7f45e8109240 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- 172.21.15.98:0/1705885957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e80ff020 0x7f45e80ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- 172.21.15.98:0/1705885957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e8101ff0 0x7f45e8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- 172.21.15.98:0/1705885957 >> 172.21.15.98:0/1705885957 conn(0x7f45e80fa950 msgr2=0x7f45e80fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- 172.21.15.98:0/1705885957 shutdown_connections 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- 172.21.15.98:0/1705885957 wait complete. 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 Processor -- start 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- start start 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 0x7f45e8101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e8102990 0x7f45e81004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8100c80 con 0x7f45e8101ff0 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8100df0 con 0x7f45e8102990 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45eeef3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81a0d50 con 0x7f45e80ff020 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.554+0000 7f45e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41294/0 (socket says 172.21.15.98:41294) 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 -- 172.21.15.98:0/3123517273 learned_addr learned my addr 172.21.15.98:0/3123517273 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45ed490700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e8102990 0x7f45e81004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45ecc8f700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 0x7f45e8101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 -- 172.21.15.98:0/3123517273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 msgr2=0x7f45e8101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 0x7f45e8101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 -- 172.21.15.98:0/3123517273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e8102990 msgr2=0x7f45e81004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e8102990 0x7f45e81004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 -- 172.21.15.98:0/3123517273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0009d00 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45ecc8f700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 0x7f45e8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e7fff700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e9a0 tx=0x7f45d400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4009790 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f45d4010430 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e81a0fe0 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4018680 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.555+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e81a14b0 con 0x7f45e8101ff0 2024-03-18T20:25:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.556+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e81082f0 con 0x7f45e8101ff0 2024-03-18T20:25:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.557+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f45d40105a0 con 0x7f45e8101ff0 2024-03-18T20:25:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.558+0000 7f45e5ffb700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 0x7f45d80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.558+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45d4014070 con 0x7f45e8101ff0 2024-03-18T20:25:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.558+0000 7f45ecc8f700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 0x7f45d80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.558+0000 7f45ecc8f700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 0x7f45d80a36f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45e8067bf0 tx=0x7f45dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.562+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f45d40529f0 con 0x7f45e8101ff0 2024-03-18T20:25:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.758+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45e802cee0 con 0x7f45e8101ff0 2024-03-18T20:25:56.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.759+0000 7f45e5ffb700 1 -- 172.21.15.98:0/3123517273 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f45d40962a0 con 0x7f45e8101ff0 2024-03-18T20:25:56.762 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 msgr2=0x7f45d80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 0x7f45d80a36f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45e8067bf0 tx=0x7f45dc009260 comp rx=0 tx=0).stop 2024-03-18T20:25:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 msgr2=0x7f45e80fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e9a0 tx=0x7f45d400ed60 comp rx=0 tx=0).stop 2024-03-18T20:25:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 shutdown_connections 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f45d80a1230 0x7f45d80a36f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45e8102990 0x7f45e81004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f45e80ff020 0x7f45e8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 --2- 172.21.15.98:0/3123517273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45e8101ff0 0x7f45e80fffa0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 >> 172.21.15.98:0/3123517273 conn(0x7f45e80fa950 msgr2=0x7f45e8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 shutdown_connections 2024-03-18T20:25:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.761+0000 7f45eeef3700 1 -- 172.21.15.98:0/3123517273 wait complete. 2024-03-18T20:25:56.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 -- 172.21.15.98:0/1377434904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c100e60 msgr2=0x7f879c1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 --2- 172.21.15.98:0/1377434904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c100e60 0x7f879c1012c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f87840099c0 tx=0x7f8784009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 -- 172.21.15.98:0/1377434904 shutdown_connections 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 --2- 172.21.15.98:0/1377434904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c101800 0x7f879c1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 --2- 172.21.15.98:0/1377434904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c0ffc70 0x7f879c100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 --2- 172.21.15.98:0/1377434904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c100e60 0x7f879c1012c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 -- 172.21.15.98:0/1377434904 >> 172.21.15.98:0/1377434904 conn(0x7f879c0fb1c0 msgr2=0x7f879c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 -- 172.21.15.98:0/1377434904 shutdown_connections 2024-03-18T20:25:56.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.898+0000 7f87a1c26700 1 -- 172.21.15.98:0/1377434904 wait complete. 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 Processor -- start 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 -- start start 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 0x7f879c195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c100e60 0x7f879c196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 0x7f879c19d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c19d620 con 0x7f879c101800 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0692f0 con 0x7f879c100e60 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.899+0000 7f87a1c26700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c069480 con 0x7f879c0ffc70 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 0x7f879c195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 0x7f879c19d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 0x7f879c195af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.98:43396/0 (socket says 172.21.15.98:43396) 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879b7fe700 1 -- 172.21.15.98:0/322911553 learned_addr learned my addr 172.21.15.98:0/322911553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879affd700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c100e60 0x7f879c196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 -- 172.21.15.98:0/322911553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 msgr2=0x7f879c195af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 0x7f879c195af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 -- 172.21.15.98:0/322911553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c100e60 msgr2=0x7f879c196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c100e60 0x7f879c196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 -- 172.21.15.98:0/322911553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87840095f0 con 0x7f879c101800 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f879bfff700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 0x7f879c19d050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f879400c970 tx=0x7f879400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879400e880 con 0x7f879c101800 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f879400ee70 con 0x7f879c101800 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c069760 con 0x7f879c101800 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8794018400 con 0x7f879c101800 2024-03-18T20:25:56.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.900+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c069c20 con 0x7f879c101800 2024-03-18T20:25:56.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.901+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c1070f0 con 0x7f879c101800 2024-03-18T20:25:56.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.902+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f879400e9e0 con 0x7f879c101800 2024-03-18T20:25:56.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.903+0000 7f8798ff9700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 0x7f87880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.903+0000 7f879b7fe700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 0x7f87880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.903+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f87940ccb10 con 0x7f879c101800 2024-03-18T20:25:56.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.903+0000 7f879b7fe700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 0x7f87880a3670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f878c009530 tx=0x7f878c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:56.907+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8794095e30 con 0x7f879c101800 2024-03-18T20:25:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:56 smithi136 ceph-mon[48073]: pgmap v262: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:56.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:56 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3483181886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:56.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:56 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3403552901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:56 smithi098 ceph-mon[43462]: pgmap v262: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:56 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3483181886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:56 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3403552901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.103+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f879c069ea0 con 0x7f879c101800 2024-03-18T20:25:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.104+0000 7f8798ff9700 1 -- 172.21.15.98:0/322911553 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8794099450 con 0x7f879c101800 2024-03-18T20:25:57.107 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:25:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 msgr2=0x7f87880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 0x7f87880a3670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f878c009530 tx=0x7f878c009040 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 msgr2=0x7f879c19d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 0x7f879c19d050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f879400c970 tx=0x7f879400cc80 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 shutdown_connections 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c101800 0x7f879c19d050 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f879c0ffc70 0x7f879c195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f87880a11b0 0x7f87880a3670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 --2- 172.21.15.98:0/322911553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879c100e60 0x7f879c196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 >> 172.21.15.98:0/322911553 conn(0x7f879c0fb1c0 msgr2=0x7f879c104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 shutdown_connections 2024-03-18T20:25:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.106+0000 7f87a1c26700 1 -- 172.21.15.98:0/322911553 wait complete. 2024-03-18T20:25:57.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:25:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:56 smithi169 ceph-mon[48529]: pgmap v262: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T20:25:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:56 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3483181886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:56 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3403552901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.243+0000 7f405cf24700 1 -- 172.21.15.98:0/3034764236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 msgr2=0x7f4058100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.243+0000 7f405cf24700 1 --2- 172.21.15.98:0/3034764236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058100080 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f404c0099c0 tx=0x7f404c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 -- 172.21.15.98:0/3034764236 shutdown_connections 2024-03-18T20:25:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 --2- 172.21.15.98:0/3034764236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f40581017f0 0x7f40581080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 --2- 172.21.15.98:0/3034764236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058100080 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 --2- 172.21.15.98:0/3034764236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4058100e50 0x7f40581012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 -- 172.21.15.98:0/3034764236 >> 172.21.15.98:0/3034764236 conn(0x7f40580fb1d0 msgr2=0x7f40580fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 -- 172.21.15.98:0/3034764236 shutdown_connections 2024-03-18T20:25:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 -- 172.21.15.98:0/3034764236 wait complete. 2024-03-18T20:25:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 Processor -- start 2024-03-18T20:25:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 -- start start 2024-03-18T20:25:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.244+0000 7f405cf24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 0x7f4058102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40581017f0 0x7f4058102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058102f10 con 0x7f40580ffc60 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058103080 con 0x7f40581017f0 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581a0c60 con 0x7f4058100e50 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41318/0 (socket says 172.21.15.98:41318) 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 -- 172.21.15.98:0/132989133 learned_addr learned my addr 172.21.15.98:0/132989133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f4055d9b700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 0x7f4058102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f4056d9d700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40581017f0 0x7f4058102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 -- 172.21.15.98:0/132989133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 msgr2=0x7f4058102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 0x7f4058102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 -- 172.21.15.98:0/132989133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40581017f0 msgr2=0x7f4058102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40581017f0 0x7f4058102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 -- 172.21.15.98:0/132989133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c0095f0 con 0x7f40580ffc60 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f4055d9b700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 0x7f4058102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405659c700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f404c009f50 tx=0x7f404c003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c01d070 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f404c00bda0 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40581a0e90 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c022e90 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.245+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40581a1200 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.246+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4058067af0 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.247+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f404c022470 con 0x7f40580ffc60 2024-03-18T20:25:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.248+0000 7f40477fe700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 0x7f40400a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.248+0000 7f4055d9b700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 0x7f40400a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.248+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f404c0ce220 con 0x7f40580ffc60 2024-03-18T20:25:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.248+0000 7f4055d9b700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 0x7f40400a3670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4048005fd0 tx=0x7f4048005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.252+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f404c0975d0 con 0x7f40580ffc60 2024-03-18T20:25:57.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.454+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f40581a13f0 con 0x7f40580ffc60 2024-03-18T20:25:57.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.455+0000 7f40477fe700 1 -- 172.21.15.98:0/132989133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f404c022720 con 0x7f40580ffc60 2024-03-18T20:25:57.458 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 msgr2=0x7f40400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 0x7f40400a3670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4048005fd0 tx=0x7f4048005e30 comp rx=0 tx=0).stop 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 msgr2=0x7f4058103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f404c009f50 tx=0x7f404c003b00 comp rx=0 tx=0).stop 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 shutdown_connections 2024-03-18T20:25:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f40400a11b0 0x7f40400a3670 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40581017f0 0x7f4058102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40580ffc60 0x7f4058103ba0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 --2- 172.21.15.98:0/132989133 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4058100e50 0x7f4058102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 >> 172.21.15.98:0/132989133 conn(0x7f40580fb1d0 msgr2=0x7f4058067ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 shutdown_connections 2024-03-18T20:25:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.457+0000 7f405cf24700 1 -- 172.21.15.98:0/132989133 wait complete. 2024-03-18T20:25:57.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 -- 172.21.15.98:0/810387795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa18101760 msgr2=0x7faa18103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 --2- 172.21.15.98:0/810387795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa18101760 0x7faa18103b50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faa08009990 tx=0x7faa08009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 -- 172.21.15.98:0/810387795 shutdown_connections 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 --2- 172.21.15.98:0/810387795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa181069c0 0x7faa18108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 --2- 172.21.15.98:0/810387795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa18101760 0x7faa18103b50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 --2- 172.21.15.98:0/810387795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 0x7faa18106480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 -- 172.21.15.98:0/810387795 >> 172.21.15.98:0/810387795 conn(0x7faa180fb0d0 msgr2=0x7faa180fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 -- 172.21.15.98:0/810387795 shutdown_connections 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.592+0000 7faa1e892700 1 -- 172.21.15.98:0/810387795 wait complete. 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 Processor -- start 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 -- start start 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 0x7faa18196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 0x7faa18197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1819e5f0 con 0x7faa181069c0 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1819e730 con 0x7faa18101760 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1e892700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1819e8c0 con 0x7faa18104090 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1ce2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1ce2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41324/0 (socket says 172.21.15.98:41324) 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa1ce2f700 1 -- 172.21.15.98:0/2856652535 learned_addr learned my addr 172.21.15.98:0/2856652535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa177fe700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 0x7faa18197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.593+0000 7faa17fff700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 0x7faa18196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 -- 172.21.15.98:0/2856652535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 msgr2=0x7faa18197000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 0x7faa18197000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 -- 172.21.15.98:0/2856652535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 msgr2=0x7faa18196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 0x7faa18196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 -- 172.21.15.98:0/2856652535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080095f0 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa17fff700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 0x7faa18196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1ce2f700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa10007ee0 tx=0x7faa1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10017070 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faa1000f040 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1819eba0 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10005710 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.594+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1819f060 con 0x7faa181069c0 2024-03-18T20:25:57.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.595+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa180fcc50 con 0x7faa181069c0 2024-03-18T20:25:57.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.596+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7faa10005870 con 0x7faa181069c0 2024-03-18T20:25:57.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.597+0000 7faa157fa700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 0x7faa000a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.597+0000 7faa17fff700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 0x7faa000a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.597+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faa100ccc20 con 0x7faa181069c0 2024-03-18T20:25:57.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.597+0000 7faa17fff700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 0x7faa000a3670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faa080095c0 tx=0x7faa08009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.600+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa10095ee0 con 0x7faa181069c0 2024-03-18T20:25:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.845+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa18067a10 con 0x7faa181069c0 2024-03-18T20:25:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.846+0000 7faa157fa700 1 -- 172.21.15.98:0/2856652535 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7faa10099500 con 0x7faa181069c0 2024-03-18T20:25:57.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:57.849 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 msgr2=0x7faa000a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 0x7faa000a3670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faa080095c0 tx=0x7faa08009510 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 msgr2=0x7faa1819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa10007ee0 tx=0x7faa1000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 shutdown_connections 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa181069c0 0x7faa1819e020 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faa000a11b0 0x7faa000a3670 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa18101760 0x7faa18196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 --2- 172.21.15.98:0/2856652535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa18104090 0x7faa18197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 >> 172.21.15.98:0/2856652535 conn(0x7faa180fb0d0 msgr2=0x7faa18103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 shutdown_connections 2024-03-18T20:25:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.848+0000 7faa1e892700 1 -- 172.21.15.98:0/2856652535 wait complete. 2024-03-18T20:25:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:57.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:57.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 -- 172.21.15.98:0/1074594560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc101e70 msgr2=0x7f43cc1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 --2- 172.21.15.98:0/1074594560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc101e70 0x7f43cc1022d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f43b40099c0 tx=0x7f43b4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 -- 172.21.15.98:0/1074594560 shutdown_connections 2024-03-18T20:25:57.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 --2- 172.21.15.98:0/1074594560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc102810 0x7f43cc1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 --2- 172.21.15.98:0/1074594560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc100c80 0x7f43cc1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 --2- 172.21.15.98:0/1074594560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc101e70 0x7f43cc1022d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.982+0000 7f43d12bc700 1 -- 172.21.15.98:0/1074594560 >> 172.21.15.98:0/1074594560 conn(0x7f43cc0fc210 msgr2=0x7f43cc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- 172.21.15.98:0/1074594560 shutdown_connections 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- 172.21.15.98:0/1074594560 wait complete. 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 Processor -- start 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- start start 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 0x7f43cc196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc102810 0x7f43cc19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19e5d0 con 0x7f43cc100c80 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19e710 con 0x7f43cc102810 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.983+0000 7f43d12bc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19e8a0 con 0x7f43cc101e70 2024-03-18T20:25:57.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41340/0 (socket says 172.21.15.98:41340) 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 -- 172.21.15.98:0/486661295 learned_addr learned my addr 172.21.15.98:0/486661295 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43ca7fc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 0x7f43cc196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 -- 172.21.15.98:0/486661295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 msgr2=0x7f43cc196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 0x7f43cc196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 -- 172.21.15.98:0/486661295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc102810 msgr2=0x7f43cc19e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43cb7fe700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc102810 0x7f43cc19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc102810 0x7f43cc19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 -- 172.21.15.98:0/486661295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40095f0 con 0x7f43cc100c80 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43ca7fc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 0x7f43cc196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43caffd700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00d980 tx=0x7f43bc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc00de60 con 0x7f43cc100c80 2024-03-18T20:25:57.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc19eb80 con 0x7f43cc100c80 2024-03-18T20:25:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f43bc0044b0 con 0x7f43cc100c80 2024-03-18T20:25:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc003b40 con 0x7f43cc100c80 2024-03-18T20:25:57.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.984+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc19f0e0 con 0x7f43cc100c80 2024-03-18T20:25:57.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3123517273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/322911553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:57 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/132989133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.985+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc108150 con 0x7f43cc100c80 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.986+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f43bc003ca0 con 0x7f43cc100c80 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.987+0000 7f43c3fff700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 0x7f43b80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.987+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f43bc0cca70 con 0x7f43cc100c80 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.987+0000 7f43ca7fc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 0x7f43b80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.987+0000 7f43ca7fc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 0x7f43b80a3670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43b4006010 tx=0x7f43b4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:57.991+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f43bc095d90 con 0x7f43cc100c80 2024-03-18T20:25:58.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3123517273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/322911553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:57 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/132989133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3123517273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/322911553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:57 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/132989133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.199+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43cc02ccb0 con 0x7f43cc100c80 2024-03-18T20:25:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.201+0000 7f43c3fff700 1 -- 172.21.15.98:0/486661295 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24488 (secure 0 0 0) 0x7f43bc009940 con 0x7f43cc100c80 2024-03-18T20:25:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:58.205 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":87,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:25:46.679841+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 msgr2=0x7f43b80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 0x7f43b80a3670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f43b4006010 tx=0x7f43b4009510 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 msgr2=0x7f43cc196aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00d980 tx=0x7f43bc00dd40 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 shutdown_connections 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f43b80a11b0 0x7f43b80a3670 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43cc102810 0x7f43cc19e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc100c80 0x7f43cc196aa0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 --2- 172.21.15.98:0/486661295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43cc101e70 0x7f43cc196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 >> 172.21.15.98:0/486661295 conn(0x7f43cc0fc210 msgr2=0x7f43cc105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 shutdown_connections 2024-03-18T20:25:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.203+0000 7f43d12bc700 1 -- 172.21.15.98:0/486661295 wait complete. 2024-03-18T20:25:58.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:25:58.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 -- 172.21.15.98:0/3798194031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4988104110 msgr2=0x7f4988106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 --2- 172.21.15.98:0/3798194031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4988104110 0x7f4988106500 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4978009990 tx=0x7f4978009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 -- 172.21.15.98:0/3798194031 shutdown_connections 2024-03-18T20:25:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 --2- 172.21.15.98:0/3798194031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988106a40 0x7f4988108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 --2- 172.21.15.98:0/3798194031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49881017e0 0x7f4988103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 --2- 172.21.15.98:0/3798194031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4988104110 0x7f4988106500 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 -- 172.21.15.98:0/3798194031 >> 172.21.15.98:0/3798194031 conn(0x7f49880fb1d0 msgr2=0x7f49880fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 -- 172.21.15.98:0/3798194031 shutdown_connections 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.341+0000 7f498e35c700 1 -- 172.21.15.98:0/3798194031 wait complete. 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 Processor -- start 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 -- start start 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988104110 0x7f4988197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 0x7f498819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819e760 con 0x7f49881017e0 2024-03-18T20:25:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819e8a0 con 0x7f4988106a40 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498e35c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819ea30 con 0x7f4988104110 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41364/0 (socket says 172.21.15.98:41364) 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 -- 172.21.15.98:0/994976136 learned_addr learned my addr 172.21.15.98:0/994976136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498c8f9700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 0x7f498819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f49877fe700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988104110 0x7f4988197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 -- 172.21.15.98:0/994976136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988104110 msgr2=0x7f4988197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988104110 0x7f4988197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 -- 172.21.15.98:0/994976136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 msgr2=0x7f498819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 0x7f498819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 -- 172.21.15.98:0/994976136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780095f0 con 0x7f49881017e0 2024-03-18T20:25:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f498c8f9700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 0x7f498819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.342+0000 7f4987fff700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f497000b530 tx=0x7f497000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.343+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970011830 con 0x7f49881017e0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.343+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4970011e20 con 0x7f49881017e0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.343+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498819ed10 con 0x7f49881017e0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.343+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970010dd0 con 0x7f49881017e0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.343+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498819f1d0 con 0x7f49881017e0 2024-03-18T20:25:58.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.344+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49880fcd50 con 0x7f49881017e0 2024-03-18T20:25:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.345+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4970011990 con 0x7f49881017e0 2024-03-18T20:25:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.345+0000 7f49857fa700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 0x7f49740a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.346+0000 7f49877fe700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 0x7f49740a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.346+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49700ccbd0 con 0x7f49881017e0 2024-03-18T20:25:58.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.346+0000 7f49877fe700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 0x7f49740a3670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4978003780 tx=0x7f4978005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.349+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4970095fa0 con 0x7f49881017e0 2024-03-18T20:25:58.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.541+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f498802cec0 con 0x7f49881017e0 2024-03-18T20:25:58.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.541+0000 7f49857fa700 1 -- 172.21.15.98:0/994976136 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49700995c0 con 0x7f49881017e0 2024-03-18T20:25:58.545 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 msgr2=0x7f49740a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 0x7f49740a3670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4978003780 tx=0x7f4978005fb0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 msgr2=0x7f4988196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f497000b530 tx=0x7f497000b8f0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 shutdown_connections 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f49740a11b0 0x7f49740a3670 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4988106a40 0x7f498819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49881017e0 0x7f4988196c30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 --2- 172.21.15.98:0/994976136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4988104110 0x7f4988197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.543+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 >> 172.21.15.98:0/994976136 conn(0x7f49880fb1d0 msgr2=0x7f4988102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.544+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 shutdown_connections 2024-03-18T20:25:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.544+0000 7f498e35c700 1 -- 172.21.15.98:0/994976136 wait complete. 2024-03-18T20:25:58.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- 172.21.15.98:0/2638052255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 msgr2=0x7f5ac41000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- 172.21.15.98:0/2638052255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac41000b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80099c0 tx=0x7f5ab8009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- 172.21.15.98:0/2638052255 shutdown_connections 2024-03-18T20:25:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- 172.21.15.98:0/2638052255 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4101820 0x7f5ac41080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- 172.21.15.98:0/2638052255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac41000b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- 172.21.15.98:0/2638052255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4100e80 0x7f5ac41012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- 172.21.15.98:0/2638052255 >> 172.21.15.98:0/2638052255 conn(0x7f5ac40fb200 msgr2=0x7f5ac40fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- 172.21.15.98:0/2638052255 shutdown_connections 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- 172.21.15.98:0/2638052255 wait complete. 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 Processor -- start 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- start start 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4100e80 0x7f5ac4197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4101820 0x7f5ac419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419e7e0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419e950 con 0x7f5ac4101820 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.681+0000 7f5acaad0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419eb10 con 0x7f5ac4100e80 2024-03-18T20:25:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41390/0 (socket says 172.21.15.98:41390) 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 -- 172.21.15.98:0/1707143014 learned_addr learned my addr 172.21.15.98:0/1707143014 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac906d700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4101820 0x7f5ac419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac3fff700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4100e80 0x7f5ac4197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 -- 172.21.15.98:0/1707143014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4100e80 msgr2=0x7f5ac4197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4100e80 0x7f5ac4197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 -- 172.21.15.98:0/1707143014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4101820 msgr2=0x7f5ac419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4101820 0x7f5ac419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 -- 172.21.15.98:0/1707143014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80095f0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac886c700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009f50 tx=0x7f5ab8005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab801d070 con 0x7f5ac40ffc90 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ab800bda0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac419ed90 con 0x7f5ac40ffc90 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8022dd0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.682+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac419f1b0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.683+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac4067260 con 0x7f5ac40ffc90 2024-03-18T20:25:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.684+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5ab8022570 con 0x7f5ac40ffc90 2024-03-18T20:25:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.685+0000 7f5ac1ffb700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 0x7f5ab40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.685+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ab80ce7d0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.685+0000 7f5ac3fff700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 0x7f5ab40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.685+0000 7f5ac3fff700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 0x7f5ab40a3670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4198170 tx=0x7f5ab000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.688+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ab8097b80 con 0x7f5ac40ffc90 2024-03-18T20:25:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:58 smithi098 ceph-mon[43462]: pgmap v263: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:25:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2856652535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:58.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/486661295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:58.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:58 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/994976136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:58.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.887+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ac419f340 con 0x7f5ac40ffc90 2024-03-18T20:25:58.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.888+0000 7f5ac1ffb700 1 -- 172.21.15.98:0/1707143014 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ab809b1a0 con 0x7f5ac40ffc90 2024-03-18T20:25:58.891 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 msgr2=0x7f5ab40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 0x7f5ab40a3670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4198170 tx=0x7f5ab000b5b0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 msgr2=0x7f5ac4196d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009f50 tx=0x7f5ab8005b00 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 shutdown_connections 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5ab40a11b0 0x7f5ab40a3670 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac4101820 0x7f5ac419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac40ffc90 0x7f5ac4196d10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 --2- 172.21.15.98:0/1707143014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ac4100e80 0x7f5ac4197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 >> 172.21.15.98:0/1707143014 conn(0x7f5ac40fb200 msgr2=0x7f5ac4069040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 shutdown_connections 2024-03-18T20:25:58.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:58.890+0000 7f5acaad0700 1 -- 172.21.15.98:0/1707143014 wait complete. 2024-03-18T20:25:58.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:58.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:58 smithi136 ceph-mon[48073]: pgmap v263: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:25:58.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2856652535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:58.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/486661295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:58.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:58 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/994976136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 -- 172.21.15.98:0/2337413236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980ff020 msgr2=0x7f42980ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 --2- 172.21.15.98:0/2337413236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980ff020 0x7f42980ff440 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f428c009990 tx=0x7f428c009ca0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 -- 172.21.15.98:0/2337413236 shutdown_connections 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 --2- 172.21.15.98:0/2337413236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f4298109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 --2- 172.21.15.98:0/2337413236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980ff020 0x7f42980ff440 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 --2- 172.21.15.98:0/2337413236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4298101ff0 0x7f4298102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 -- 172.21.15.98:0/2337413236 >> 172.21.15.98:0/2337413236 conn(0x7f42980fa950 msgr2=0x7f42980fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 -- 172.21.15.98:0/2337413236 shutdown_connections 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 -- 172.21.15.98:0/2337413236 wait complete. 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.027+0000 7f429ee2d700 1 Processor -- start 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 -- start start 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42980ff020 0x7f4298196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f429819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429819e630 con 0x7f4298101ff0 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429819e770 con 0x7f4298102990 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429819e900 con 0x7f42980ff020 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41398/0 (socket says 172.21.15.98:41398) 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 -- 172.21.15.98:0/3725158316 learned_addr learned my addr 172.21.15.98:0/3725158316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429d3ca700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f429819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429cbc9700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42980ff020 0x7f4298196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 -- 172.21.15.98:0/3725158316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42980ff020 msgr2=0x7f4298196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42980ff020 0x7f4298196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 -- 172.21.15.98:0/3725158316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 msgr2=0x7f429819e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f429819e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 -- 172.21.15.98:0/3725158316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c0095f0 con 0x7f4298101ff0 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429d3ca700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f429819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:59.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4297fff700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f428400c9b0 tx=0x7f428400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284007b00 con 0x7f4298101ff0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4284007c60 con 0x7f4298101ff0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429819ebe0 con 0x7f4298101ff0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.028+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284018660 con 0x7f4298101ff0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.029+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429819f0f0 con 0x7f4298101ff0 2024-03-18T20:25:59.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.030+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298108230 con 0x7f4298101ff0 2024-03-18T20:25:59.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.031+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f428400f450 con 0x7f4298101ff0 2024-03-18T20:25:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.031+0000 7f4295ffb700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 0x7f42880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.031+0000 7f429cbc9700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 0x7f42880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.031+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f42840ccdd0 con 0x7f4298101ff0 2024-03-18T20:25:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.032+0000 7f429cbc9700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 0x7f42880a3670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f428c00b5c0 tx=0x7f428c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.035+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f42840960f0 con 0x7f4298101ff0 2024-03-18T20:25:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:58 smithi169 ceph-mon[48529]: pgmap v263: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:25:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2856652535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:25:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/486661295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:25:59.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:58 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/994976136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:25:59.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.230+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f429802d120 con 0x7f4298101ff0 2024-03-18T20:25:59.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.231+0000 7f4295ffb700 1 -- 172.21.15.98:0/3725158316 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4284099710 con 0x7f4298101ff0 2024-03-18T20:25:59.235 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 msgr2=0x7f42880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 0x7f42880a3670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f428c00b5c0 tx=0x7f428c005fd0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 msgr2=0x7f4298197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f428400c9b0 tx=0x7f428400cd70 comp rx=0 tx=0).stop 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 shutdown_connections 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42880a11b0 0x7f42880a3670 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4298102990 0x7f429819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f42980ff020 0x7f4298196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 --2- 172.21.15.98:0/3725158316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4298101ff0 0x7f4298197040 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 >> 172.21.15.98:0/3725158316 conn(0x7f42980fa950 msgr2=0x7f4298105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 shutdown_connections 2024-03-18T20:25:59.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.233+0000 7f429ee2d700 1 -- 172.21.15.98:0/3725158316 wait complete. 2024-03-18T20:25:59.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:25:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 -- 172.21.15.98:0/2834977818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 msgr2=0x7fa3241012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 --2- 172.21.15.98:0/2834977818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa3241012b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa3140099c0 tx=0x7fa314009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 -- 172.21.15.98:0/2834977818 shutdown_connections 2024-03-18T20:25:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 --2- 172.21.15.98:0/2834977818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3241017f0 0x7fa3241080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 --2- 172.21.15.98:0/2834977818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3240ffc60 0x7fa324100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 --2- 172.21.15.98:0/2834977818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa3241012b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 -- 172.21.15.98:0/2834977818 >> 172.21.15.98:0/2834977818 conn(0x7fa3240fb1d0 msgr2=0x7fa3240fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 -- 172.21.15.98:0/2834977818 shutdown_connections 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.372+0000 7fa328f65700 1 -- 172.21.15.98:0/2834977818 wait complete. 2024-03-18T20:25:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 Processor -- start 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 -- start start 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3240ffc60 0x7fa324103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 0x7fa324102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa324102f10 con 0x7fa324100e50 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa324103080 con 0x7fa3241017f0 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3241a0cb0 con 0x7fa3240ffc60 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41414/0 (socket says 172.21.15.98:41414) 2024-03-18T20:25:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 -- 172.21.15.98:0/204180564 learned_addr learned my addr 172.21.15.98:0/204180564 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa32259c700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3240ffc60 0x7fa324103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa322d9d700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 0x7fa324102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 -- 172.21.15.98:0/204180564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3240ffc60 msgr2=0x7fa324103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3240ffc60 0x7fa324103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 -- 172.21.15.98:0/204180564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 msgr2=0x7fa324102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 0x7fa324102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 -- 172.21.15.98:0/204180564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140095f0 con 0x7fa324100e50 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa322d9d700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 0x7fa324102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa321d9b700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa314000c00 tx=0x7fa31400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31401d070 con 0x7fa324100e50 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa3140054c0 con 0x7fa324100e50 2024-03-18T20:25:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3241a0ee0 con 0x7fa324100e50 2024-03-18T20:25:59.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.373+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314004c70 con 0x7fa324100e50 2024-03-18T20:25:59.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.374+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3241a12a0 con 0x7fa324100e50 2024-03-18T20:25:59.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.374+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa324067af0 con 0x7fa324100e50 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.375+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa314004dd0 con 0x7fa324100e50 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.376+0000 7fa3137fe700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 0x7fa30c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.376+0000 7fa32259c700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 0x7fa30c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.376+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa3140cdb50 con 0x7fa324100e50 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.376+0000 7fa32259c700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 0x7fa30c0a36f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa324100cb0 tx=0x7fa318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.380+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa314096f00 con 0x7fa324100e50 2024-03-18T20:25:59.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.578+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa3241a1490 con 0x7fa324100e50 2024-03-18T20:25:59.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.578+0000 7fa3137fe700 1 -- 172.21.15.98:0/204180564 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa31409a520 con 0x7fa324100e50 2024-03-18T20:25:59.582 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 msgr2=0x7fa30c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 0x7fa30c0a36f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa324100cb0 tx=0x7fa318009260 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 msgr2=0x7fa324102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa314000c00 tx=0x7fa31400f740 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 shutdown_connections 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fa30c0a1230 0x7fa30c0a36f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3241017f0 0x7fa324102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3240ffc60 0x7fa324103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 --2- 172.21.15.98:0/204180564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa324100e50 0x7fa324102230 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 >> 172.21.15.98:0/204180564 conn(0x7fa3240fb1d0 msgr2=0x7fa324067ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 shutdown_connections 2024-03-18T20:25:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.580+0000 7fa328f65700 1 -- 172.21.15.98:0/204180564 wait complete. 2024-03-18T20:25:59.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 -- 172.21.15.98:0/2727963739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 msgr2=0x7ff0100fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 --2- 172.21.15.98:0/2727963739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff0100fff80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff00c0099c0 tx=0x7ff00c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 -- 172.21.15.98:0/2727963739 shutdown_connections 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 --2- 172.21.15.98:0/2727963739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff010107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 --2- 172.21.15.98:0/2727963739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff0100fff80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 --2- 172.21.15.98:0/2727963739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 0x7ff0101011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 -- 172.21.15.98:0/2727963739 >> 172.21.15.98:0/2727963739 conn(0x7ff0100fb0d0 msgr2=0x7ff0100fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.715+0000 7ff0176cc700 1 -- 172.21.15.98:0/2727963739 shutdown_connections 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 -- 172.21.15.98:0/2727963739 wait complete. 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 Processor -- start 2024-03-18T20:25:59.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 -- start start 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 0x7ff01019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff01019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01019e5e0 con 0x7ff0100ffb60 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01019e750 con 0x7ff010100d50 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff0176cc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01019e910 con 0x7ff0101016f0 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff015468700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff015468700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41438/0 (socket says 172.21.15.98:41438) 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.716+0000 7ff015468700 1 -- 172.21.15.98:0/880104519 learned_addr learned my addr 172.21.15.98:0/880104519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015c69700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff01019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 -- 172.21.15.98:0/880104519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 msgr2=0x7ff01019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff01019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 -- 172.21.15.98:0/880104519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 msgr2=0x7ff01019b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff014c67700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 0x7ff01019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 0x7ff01019b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 -- 172.21.15.98:0/880104519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c0095f0 con 0x7ff0100ffb60 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015c69700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff01019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff015468700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff00c0098e0 tx=0x7ff00c003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c01d070 con 0x7ff0100ffb60 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff00c00bd50 con 0x7ff0100ffb60 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c022d90 con 0x7ff0100ffb60 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01019eb30 con 0x7ff0100ffb60 2024-03-18T20:25:59.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.717+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01019efe0 con 0x7ff0100ffb60 2024-03-18T20:25:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.718+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff010107050 con 0x7ff0100ffb60 2024-03-18T20:25:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.719+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7ff00c022470 con 0x7ff0100ffb60 2024-03-18T20:25:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.719+0000 7ff0027fc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 0x7feffc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.720+0000 7ff014c67700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 0x7feffc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.720+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff00c0ce8e0 con 0x7ff0100ffb60 2024-03-18T20:25:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.720+0000 7ff014c67700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 0x7feffc0a3770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff010068ed0 tx=0x7ff004005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.723+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff00c097d40 con 0x7ff0100ffb60 2024-03-18T20:25:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.966+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff010067a10 con 0x7ff0100ffb60 2024-03-18T20:25:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.967+0000 7ff0027fc700 1 -- 172.21.15.98:0/880104519 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10875 (secure 0 0 0) 0x7ff00c09b360 con 0x7ff0100ffb60 2024-03-18T20:25:59.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:25:59.971 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/2703259381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2703259381},{"type":"v1","addr":"172.21.15.98:6835","nonce":2703259381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1932954082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1932954082},{"type":"v1","addr":"172.21.15.98:6837","nonce":1932954082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/655758721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":655758721},{"type":"v1","addr":"172.21.15.98:6839","nonce":655758721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/2638633847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":2638633847},{"type":"v1","addr":"172.21.15.98:6841","nonce":2638633847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6833/1320002530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":1320002530},{"type":"v1","addr":"172.21.15.169:6833","nonce":1320002530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1211632092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1211632092},{"type":"v1","addr":"172.21.15.136:6833","nonce":1211632092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T20:24:41.335739+0000","modified":"2024-03-18T20:25:00.478029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24494,"mds_1":24475,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/4136937912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4136937912},{"type":"v1","addr":"172.21.15.136:6835","nonce":4136937912}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6835/477760816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":477760816},{"type":"v1","addr":"172.21.15.169:6835","nonce":477760816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.169:6837/1340440313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":1340440313},{"type":"v1","addr":"172.21.15.169:6837","nonce":1340440313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.136:6837/3347718998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3347718998},{"type":"v1","addr":"172.21.15.136:6837","nonce":3347718998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.169:6839/2458510608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":2458510608},{"type":"v1","addr":"172.21.15.169:6839","nonce":2458510608}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.136:6839/2871006308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2871006308},{"type":"v1","addr":"172.21.15.136:6839","nonce":2871006308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 msgr2=0x7feffc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 0x7feffc0a3770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff010068ed0 tx=0x7ff004005cf0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 msgr2=0x7ff010068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff00c0098e0 tx=0x7ff00c003b80 comp rx=0 tx=0).stop 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 shutdown_connections 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff0101016f0 0x7ff01019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0100ffb60 0x7ff010068930 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7feffc0a12b0 0x7feffc0a3770 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 --2- 172.21.15.98:0/880104519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff010100d50 0x7ff01019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 >> 172.21.15.98:0/880104519 conn(0x7ff0100fb0d0 msgr2=0x7ff010104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 shutdown_connections 2024-03-18T20:25:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:25:59.969+0000 7ff0176cc700 1 -- 172.21.15.98:0/880104519 wait complete. 2024-03-18T20:25:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-18T20:25:59.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1707143014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:25:59.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3725158316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:25:59.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:25:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/204180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:25:59.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/827293782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4101e80 msgr2=0x7fd4f41022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/827293782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4101e80 0x7fd4f41022e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40099c0 tx=0x7fd4e4009cd0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/827293782 shutdown_connections 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/827293782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f4102820 0x7fd4f41090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/827293782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f4100c90 0x7fd4f41010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/827293782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4101e80 0x7fd4f41022e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/827293782 >> 172.21.15.98:0/827293782 conn(0x7fd4f40fc200 msgr2=0x7fd4f40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/827293782 shutdown_connections 2024-03-18T20:26:00.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.103+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/827293782 wait complete. 2024-03-18T20:26:00.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 Processor -- start 2024-03-18T20:26:00.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 -- start start 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f4101e80 0x7fd4f4197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f4102820 0x7fd4f419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f419e620 con 0x7fd4f4100c90 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f419e760 con 0x7fd4f4101e80 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f8bb6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f419e8f0 con 0x7fd4f4102820 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41448/0 (socket says 172.21.15.98:41448) 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 -- 172.21.15.98:0/2452435698 learned_addr learned my addr 172.21.15.98:0/2452435698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 -- 172.21.15.98:0/2452435698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f4102820 msgr2=0x7fd4f419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f4102820 0x7fd4f419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 -- 172.21.15.98:0/2452435698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f4101e80 msgr2=0x7fd4f4197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f4101e80 0x7fd4f4197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.104+0000 7fd4f259c700 1 -- 172.21.15.98:0/2452435698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40095f0 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4f259c700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00d980 tx=0x7fd4dc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00de60 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f419ebd0 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4dc0044b0 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc003b40 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.105+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f419f0e0 con 0x7fd4f4100c90 2024-03-18T20:26:00.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.106+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f4108100 con 0x7fd4f4100c90 2024-03-18T20:26:00.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.107+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd4dc004870 con 0x7fd4f4100c90 2024-03-18T20:26:00.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.107+0000 7fd4eb7fe700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 0x7fd4e00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.107+0000 7fd4f1d9b700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 0x7fd4e00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.108+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4dc0cce90 con 0x7fd4f4100c90 2024-03-18T20:26:00.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.108+0000 7fd4f1d9b700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 0x7fd4e00a3670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40098e0 tx=0x7fd4e4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.111+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4dc096150 con 0x7fd4f4100c90 2024-03-18T20:26:00.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1707143014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3725158316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:25:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/204180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1707143014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3725158316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:25:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/204180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:26:00.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.319+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4f402ce40 con 0x7fd4f4100c90 2024-03-18T20:26:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.320+0000 7fd4eb7fe700 1 -- 172.21.15.98:0/2452435698 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24488 (secure 0 0 0) 0x7fd4dc009940 con 0x7fd4f4100c90 2024-03-18T20:26:00.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:26:00.324 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":87,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:25:46.679841+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 msgr2=0x7fd4e00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 0x7fd4e00a3670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40098e0 tx=0x7fd4e4000bc0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 msgr2=0x7fd4f4196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00d980 tx=0x7fd4dc00dd40 comp rx=0 tx=0).stop 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 shutdown_connections 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4f4102820 0x7fd4f419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4f4100c90 0x7fd4f4196af0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fd4e00a11b0 0x7fd4e00a3670 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 --2- 172.21.15.98:0/2452435698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f4101e80 0x7fd4f4197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 >> 172.21.15.98:0/2452435698 conn(0x7fd4f40fc200 msgr2=0x7fd4f4105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.322+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 shutdown_connections 2024-03-18T20:26:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.323+0000 7fd4f8bb6700 1 -- 172.21.15.98:0/2452435698 wait complete. 2024-03-18T20:26:00.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T20:24:41.335739+0000', 'modified': '2024-03-18T20:25:00.478029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24494, 'mds_1': 24475, 'mds_2': 24461}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6835/4136937912', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 4136937912}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 4136937912}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6835/477760816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6834', 'nonce': 477760816}, {'type': 'v1', 'addr': '172.21.15.169:6835', 'nonce': 477760816}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.169:6837/1340440313', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6836', 'nonce': 1340440313}, {'type': 'v1', 'addr': '172.21.15.169:6837', 'nonce': 1340440313}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.136:6837/3347718998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3347718998}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3347718998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.169:6839/2458510608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6838', 'nonce': 2458510608}, {'type': 'v1', 'addr': '172.21.15.169:6839', 'nonce': 2458510608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.136:6839/2871006308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 2871006308}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 2871006308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:26:00.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T20:26:00.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T20:26:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 -- 172.21.15.98:0/735025283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 msgr2=0x7f75fc1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 --2- 172.21.15.98:0/735025283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc1080b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f75f400b0d0 tx=0x7f75f400b3e0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 -- 172.21.15.98:0/735025283 shutdown_connections 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 --2- 172.21.15.98:0/735025283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc1080b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 --2- 172.21.15.98:0/735025283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75fc0ffc70 0x7f75fc100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 --2- 172.21.15.98:0/735025283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc100e60 0x7f75fc1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 -- 172.21.15.98:0/735025283 >> 172.21.15.98:0/735025283 conn(0x7f75fc0fb200 msgr2=0x7f75fc0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 -- 172.21.15.98:0/735025283 shutdown_connections 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 -- 172.21.15.98:0/735025283 wait complete. 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.460+0000 7f760261e700 1 Processor -- start 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 -- start start 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc0ffc70 0x7f75fc103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75fc100e60 0x7f75fc102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc102f20 con 0x7f75fc101800 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc103090 con 0x7f75fc0ffc70 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f760261e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc1a0d80 con 0x7f75fc100e60 2024-03-18T20:26:00.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41466/0 (socket says 172.21.15.98:41466) 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 -- 172.21.15.98:0/486742766 learned_addr learned my addr 172.21.15.98:0/486742766 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f75fbfff700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc0ffc70 0x7f75fc103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 -- 172.21.15.98:0/486742766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75fc100e60 msgr2=0x7f75fc102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75fc100e60 0x7f75fc102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 -- 172.21.15.98:0/486742766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc0ffc70 msgr2=0x7f75fc103bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc0ffc70 0x7f75fc103bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 -- 172.21.15.98:0/486742766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4009d00 con 0x7f75fc101800 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.461+0000 7f7600bbb700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f75f4007b60 tx=0x7f75f40094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.462+0000 7f75f97fa700 1 -- 172.21.15.98:0/486742766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f400e040 con 0x7f75fc101800 2024-03-18T20:26:00.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.462+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc1a0fb0 con 0x7f75fc101800 2024-03-18T20:26:00.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.462+0000 7f75f97fa700 1 -- 172.21.15.98:0/486742766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75f4012d90 con 0x7f75fc101800 2024-03-18T20:26:00.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.462+0000 7f75f97fa700 1 -- 172.21.15.98:0/486742766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f401de60 con 0x7f75fc101800 2024-03-18T20:26:00.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.462+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc1a1370 con 0x7f75fc101800 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.464+0000 7f75f97fa700 1 -- 172.21.15.98:0/486742766 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f75f4012900 con 0x7f75fc101800 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.464+0000 7f75f97fa700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 0x7f75e40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f75fbfff700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 0x7f75e40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f75f97fa700 1 -- 172.21.15.98:0/486742766 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75f40cec60 con 0x7f75fc101800 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f75fbfff700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 0x7f75e40a3670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f75fc100cc0 tx=0x7f75ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f760e7b3b80 1 Processor -- start 2024-03-18T20:26:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f760e7b3b80 1 -- start start 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.465+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dc6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dc7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43d1e940 con 0x55ec43dc5bd0 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43d1c300 con 0x55ec43dc73d0 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43d17bf0 con 0x55ec43dc6530 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41470/0 (socket says 172.21.15.98:41470) 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 -- 172.21.15.98:0/263475728 learned_addr learned my addr 172.21.15.98:0/263475728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fb7fe700 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dc6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f7600bbb700 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dc7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 -- 172.21.15.98:0/263475728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 msgr2=0x55ec43dc6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dc6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 -- 172.21.15.98:0/263475728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 msgr2=0x55ec43dc7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dc7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 -- 172.21.15.98:0/263475728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec43b99360 con 0x55ec43dc5bd0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75fbfff700 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f75ec010f70 tx=0x7f75ec013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d11b88f5893243a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75e27fc700 1 -- 172.21.15.98:0/263475728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec011db0 con 0x55ec43dc5bd0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75e27fc700 1 -- 172.21.15.98:0/263475728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75ec015070 con 0x55ec43dc5bd0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.466+0000 7f75e27fc700 1 -- 172.21.15.98:0/263475728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec01c610 con 0x55ec43dc5bd0 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- 172.21.15.98:0/263475728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 msgr2=0x55ec43dc5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f75ec010f70 tx=0x7f75ec013000 comp rx=0 tx=0).stop 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- 172.21.15.98:0/263475728 shutdown_connections 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dc7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dc5ff0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/263475728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dc6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- 172.21.15.98:0/263475728 >> 172.21.15.98:0/263475728 conn(0x55ec43d3b1d0 msgr2=0x55ec43dc0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- 172.21.15.98:0/263475728 shutdown_connections 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- 172.21.15.98:0/263475728 wait complete. 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.467+0000 7f760e7b3b80 1 -- start start 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 Processor -- start 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dcff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43cb3d30 con 0x55ec43dc5bd0 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43cb39d0 con 0x55ec43dc73d0 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43b9b650 con 0x55ec43dc6530 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f760e7b3b80 10 client.0 _ll_register_callbacks cb 0x55ec43b1f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41472/0 (socket says 172.21.15.98:41472) 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 -- 172.21.15.98:0/3892054289 learned_addr learned my addr 172.21.15.98:0/3892054289 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fb7fe700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dcff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 -- 172.21.15.98:0/3892054289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 msgr2=0x55ec43dcff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dcff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 -- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 msgr2=0x55ec43dd6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 -- 172.21.15.98:0/3892054289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec010c50 con 0x55ec43dc5bd0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f75ec010a40 tx=0x7f75ec011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f455a18d726fe82a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.468+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec02c070 con 0x55ec43dc5bd0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75ec017070 con 0x55ec43dc5bd0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec43db8a80 con 0x55ec43dc5bd0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75ec01e630 con 0x55ec43dc5bd0 2024-03-18T20:26:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 10 client.15357 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ec43b45b10 con 0x55ec43dc5bd0 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55ec43db8a80 con 0x55ec43dc5bd0 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75ec026c80 con 0x55ec43dc5bd0 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 10 client.15357 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 10 client.15357 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 20 client.15357 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43ddab00 0x55ec43ddfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 4 client.15357 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 --> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec43d106b0 con 0x55ec43ddab00 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.469+0000 7f75fb7fe700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43ddab00 0x55ec43ddfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.470+0000 7f75db7fe700 10 client.15357 ms_handle_connect on v2:172.21.15.136:6836/3347718998 2024-03-18T20:26:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.470+0000 7f75fb7fe700 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43ddab00 0x55ec43ddfe20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=138e329eeb0254ac server_cookie=b8890a1ef7cd9470 in_seq=0 out_seq=0 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.470+0000 7f75db7fe700 1 -- 172.21.15.98:0/3892054289 <== mds.2 v2:172.21.15.136:6836/3347718998 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec43d106b0 con 0x55ec43ddab00 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.470+0000 7f75db7fe700 10 client.15357 handle_command_reply: tid=0 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.470+0000 7f760e7b3b80 1 client.15357 shutdown 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 msgr2=0x55ec43dcfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f75ec010a40 tx=0x7f75ec011e70 comp rx=0 tx=0).stop 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 shutdown_connections 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43ddab00 0x55ec43ddfe20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcfa30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3892054289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dcff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f75db7fe700 0 client.15357 ms_handle_reset on v2:172.21.15.136:6836/3347718998 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 >> 172.21.15.98:0/3892054289 conn(0x55ec43d3b1d0 msgr2=0x55ec43dc01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 shutdown_connections 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3892054289 wait complete. 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.471+0000 7f760e7b3b80 20 client.15357 trim_cache size 0 max 16384 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.488+0000 7f760e7b3b80 1 Processor -- start 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.488+0000 7f760e7b3b80 1 -- start start 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc6530 0x55ec43dca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43d106b0 con 0x55ec43dc5bd0 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43cd5ad0 con 0x55ec43dc6530 2024-03-18T20:26:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43d10990 con 0x55ec43dc73d0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41474/0 (socket says 172.21.15.98:41474) 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 -- 172.21.15.98:0/3664307482 learned_addr learned my addr 172.21.15.98:0/3664307482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 -- 172.21.15.98:0/3664307482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc73d0 msgr2=0x55ec43dd6f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 -- 172.21.15.98:0/3664307482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc6530 msgr2=0x55ec43dca810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc6530 0x55ec43dca810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 -- 172.21.15.98:0/3664307482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec43cdde50 con 0x55ec43dc5bd0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75fbfff700 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75ec005f80 tx=0x7f75ec011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216f39d294be13f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75e2ffd700 1 -- 172.21.15.98:0/3664307482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec02c030 con 0x55ec43dc5bd0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75e2ffd700 1 -- 172.21.15.98:0/3664307482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75ec01a6e0 con 0x55ec43dc5bd0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.489+0000 7f75e2ffd700 1 -- 172.21.15.98:0/3664307482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec01ae70 con 0x55ec43dc5bd0 2024-03-18T20:26:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3664307482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 msgr2=0x55ec43dca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75ec005f80 tx=0x7f75ec011a00 comp rx=0 tx=0).stop 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3664307482 shutdown_connections 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc73d0 0x55ec43dd6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc5bd0 0x55ec43dca2d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/3664307482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc6530 0x55ec43dca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3664307482 >> 172.21.15.98:0/3664307482 conn(0x55ec43d3b1d0 msgr2=0x55ec43dcd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3664307482 shutdown_connections 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- 172.21.15.98:0/3664307482 wait complete. 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- start start 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 Processor -- start 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dd6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43e2f410 con 0x55ec43dc73d0 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43e2f580 con 0x55ec43dc5bd0 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.490+0000 7f760e7b3b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec43e2f740 con 0x55ec43dc6530 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f760e7b3b80 10 client.0 _ll_register_callbacks cb 0x7f75f4006040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41482/0 (socket says 172.21.15.98:41482) 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 -- 172.21.15.98:0/2104118239 learned_addr learned my addr 172.21.15.98:0/2104118239 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75fbfff700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 -- 172.21.15.98:0/2104118239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 msgr2=0x55ec43dd6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dd6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 -- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc5bd0 msgr2=0x55ec43dcbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 -- 172.21.15.98:0/2104118239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec43b45b10 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f7600bbb700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009cd0 tx=0x7f75f409b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f289a3328a7cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4012bb0 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75f409c370 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75ec015070 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75f409bc70 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f760e7b3b80 10 client.15363 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ec43b9dc80 con 0x55ec43dc73d0 2024-03-18T20:26:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.491+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75f4065020 con 0x55ec43dc73d0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75f409bed0 con 0x55ec43dc73d0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 10 client.15363 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 10 client.15363 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 20 client.15363 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43e342e0 0x55ec43e30c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 4 client.15363 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 --> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec43b66ca0 con 0x55ec43e342e0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75fbfff700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43e342e0 0x55ec43e30c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75db7fe700 10 client.15363 ms_handle_connect on v2:172.21.15.136:6836/3347718998 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75fbfff700 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43e342e0 0x55ec43e30c60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=119603d059511f76 server_cookie=296db0629af811c4 in_seq=0 out_seq=0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75db7fe700 1 -- 172.21.15.98:0/2104118239 <== mds.2 v2:172.21.15.136:6836/3347718998 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ec43b66ca0 con 0x55ec43e342e0 2024-03-18T20:26:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f75db7fe700 10 client.15363 handle_command_reply: tid=0 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.492+0000 7f760e7b3b80 1 client.15363 shutdown 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 msgr2=0x55ec43dcc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009cd0 tx=0x7f75f409b5d0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 shutdown_connections 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:6836/3347718998,v1:172.21.15.136:6837/3347718998] conn(0x55ec43e342e0 0x55ec43e30c60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ec43dc73d0 0x55ec43dcc630 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f75db7fe700 0 client.15363 ms_handle_reset on v2:172.21.15.136:6836/3347718998 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ec43dc5bd0 0x55ec43dcbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 --2- 172.21.15.98:0/2104118239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec43dc6530 0x55ec43dd6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 >> 172.21.15.98:0/2104118239 conn(0x55ec43d3b1d0 msgr2=0x55ec43dcd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 shutdown_connections 2024-03-18T20:26:00.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 1 -- 172.21.15.98:0/2104118239 wait complete. 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760e7b3b80 20 client.15363 trim_cache size 0 max 16384 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:26:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 61.024428696000001, 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 87, 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 91.722632125000004 2024-03-18T20:26:00.514 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:26:00.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 msgr2=0x7f75e40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 0x7f75e40a3670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f75fc100cc0 tx=0x7f75ec00b410 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 msgr2=0x7f75fc102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f75f4007b60 tx=0x7f75f40094c0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 shutdown_connections 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75fc101800 0x7f75fc102780 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f75e40a11b0 0x7f75e40a3670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75fc0ffc70 0x7f75fc103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 --2- 172.21.15.98:0/486742766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75fc100e60 0x7f75fc102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 >> 172.21.15.98:0/486742766 conn(0x7f75fc0fb200 msgr2=0x7f75fc069060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.493+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 shutdown_connections 2024-03-18T20:26:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.494+0000 7f760261e700 1 -- 172.21.15.98:0/486742766 wait complete. 2024-03-18T20:26:00.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T20:26:00.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 -- 172.21.15.98:0/725895071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae0104110 msgr2=0x7faae0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 --2- 172.21.15.98:0/725895071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae0104110 0x7faae0106500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faad4009990 tx=0x7faad4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 -- 172.21.15.98:0/725895071 shutdown_connections 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 --2- 172.21.15.98:0/725895071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 0x7faae0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 --2- 172.21.15.98:0/725895071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae01017e0 0x7faae0103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 --2- 172.21.15.98:0/725895071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae0104110 0x7faae0106500 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 -- 172.21.15.98:0/725895071 >> 172.21.15.98:0/725895071 conn(0x7faae00fb1d0 msgr2=0x7faae00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 -- 172.21.15.98:0/725895071 shutdown_connections 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 -- 172.21.15.98:0/725895071 wait complete. 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.638+0000 7faae8505700 1 Processor -- start 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 -- start start 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae0104110 0x7faae0197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 0x7faae019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019e710 con 0x7faae01017e0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019e850 con 0x7faae0106a40 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019e9e0 con 0x7faae0104110 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41500/0 (socket says 172.21.15.98:41500) 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 -- 172.21.15.98:0/3203038172 learned_addr learned my addr 172.21.15.98:0/3203038172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae0104110 0x7faae0197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae6aa2700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 0x7faae019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 -- 172.21.15.98:0/3203038172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae0104110 msgr2=0x7faae0197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae0104110 0x7faae0197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 -- 172.21.15.98:0/3203038172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 msgr2=0x7faae019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 0x7faae019e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 -- 172.21.15.98:0/3203038172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40095f0 con 0x7faae01017e0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae62a1700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faadc00d950 tx=0x7faadc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faad37fe700 1 -- 172.21.15.98:0/3203038172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00f890 con 0x7faae01017e0 2024-03-18T20:26:00.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faad37fe700 1 -- 172.21.15.98:0/3203038172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faadc00fe80 con 0x7faae01017e0 2024-03-18T20:26:00.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faad37fe700 1 -- 172.21.15.98:0/3203038172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00e3b0 con 0x7faae01017e0 2024-03-18T20:26:00.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.639+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae019ecc0 con 0x7faae01017e0 2024-03-18T20:26:00.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.640+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae019f180 con 0x7faae01017e0 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.642+0000 7faad37fe700 1 -- 172.21.15.98:0/3203038172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7faadc010460 con 0x7faae01017e0 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.642+0000 7faad37fe700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 0x7faacc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.642+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 0x7faacc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.642+0000 7faad37fe700 1 -- 172.21.15.98:0/3203038172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faadc0ccb20 con 0x7faae01017e0 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 0x7faacc0a3670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faad4000c00 tx=0x7faad4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 Processor -- start 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 -- start start 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3b9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3bac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a30d940 con 0x56357a3b9940 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a30b300 con 0x56357a3ba7e0 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faaf469ab80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a306bf0 con 0x56357a3b8fe0 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41510/0 (socket says 172.21.15.98:41510) 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.643+0000 7faae5aa0700 1 -- 172.21.15.98:0/1859983279 learned_addr learned my addr 172.21.15.98:0/1859983279 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae62a1700 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 -- 172.21.15.98:0/1859983279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 msgr2=0x56357a3b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 -- 172.21.15.98:0/1859983279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 msgr2=0x56357a3bac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3bac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 -- 172.21.15.98:0/1859983279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56357a188360 con 0x56357a3b9940 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faae5aa0700 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faad4015000 tx=0x7faad4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd952e88c415d23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faad0ff9700 1 -- 172.21.15.98:0/1859983279 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad402b070 con 0x56357a3b9940 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faad0ff9700 1 -- 172.21.15.98:0/1859983279 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faad401e040 con 0x56357a3b9940 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.644+0000 7faad0ff9700 1 -- 172.21.15.98:0/1859983279 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4004ea0 con 0x56357a3b9940 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- 172.21.15.98:0/1859983279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 msgr2=0x56357a3ba200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faad4015000 tx=0x7faad4020000 comp rx=0 tx=0).stop 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- 172.21.15.98:0/1859983279 shutdown_connections 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3bac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 --2- 172.21.15.98:0/1859983279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ba200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- 172.21.15.98:0/1859983279 >> 172.21.15.98:0/1859983279 conn(0x56357a32a1d0 msgr2=0x56357a3afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- 172.21.15.98:0/1859983279 shutdown_connections 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- 172.21.15.98:0/1859983279 wait complete. 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 -- start start 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 Processor -- start 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.645+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b9940 0x56357a3c3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a2a2d30 con 0x56357a3ba7e0 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a2a29d0 con 0x56357a3b9940 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a18a650 con 0x56357a3b8fe0 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 10 client.0 _ll_register_callbacks cb 0x56357a10e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41516/0 (socket says 172.21.15.98:41516) 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 -- 172.21.15.98:0/2602010309 learned_addr learned my addr 172.21.15.98:0/2602010309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae5aa0700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b9940 0x56357a3c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae62a1700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 -- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 msgr2=0x56357a3c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 -- 172.21.15.98:0/2602010309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b9940 msgr2=0x56357a3c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b9940 0x56357a3c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 -- 172.21.15.98:0/2602010309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae0109aa0 con 0x56357a3ba7e0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae62a1700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faae6aa2700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faad8006070 tx=0x7faad8004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708c16ec383b1fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faac9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8003a90 con 0x56357a3ba7e0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faad8007910 con 0x56357a3ba7e0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56357a3a7a80 con 0x56357a3ba7e0 2024-03-18T20:26:00.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad8007ad0 con 0x56357a3ba7e0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 10 client.15375 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.646+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56357a134b10 con 0x56357a3ba7e0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x56357a3a7a80 con 0x56357a3ba7e0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7faad80143e0 con 0x56357a3ba7e0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 10 client.15375 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 10 client.15375 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 20 client.15375 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a3cd5f0 0x56357a3d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 4 client.15375 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 --> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56357a2ff6b0 con 0x56357a3cd5f0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.647+0000 7faae62a1700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a3cd5f0 0x56357a3d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faac9ffb700 10 client.15375 ms_handle_connect on v2:172.21.15.169:6838/2458510608 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faae62a1700 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a3cd5f0 0x56357a3d33d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c126406e682da3f server_cookie=4442d3ac6da38055 in_seq=0 out_seq=0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faac9ffb700 1 -- 172.21.15.98:0/2602010309 <== mds.1 v2:172.21.15.169:6838/2458510608 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56357a2ff6b0 con 0x56357a3cd5f0 2024-03-18T20:26:00.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faac9ffb700 10 client.15375 handle_command_reply: tid=0 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 client.15375 shutdown 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 msgr2=0x56357a3ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faad8006070 tx=0x7faad8004320 comp rx=0 tx=0).stop 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 shutdown_connections 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a3cd5f0 0x56357a3d33d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3ba7e0 0x56357a3ca3a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faac9ffb700 0 client.15375 ms_handle_reset on v2:172.21.15.169:6838/2458510608 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2602010309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b9940 0x56357a3c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 >> 172.21.15.98:0/2602010309 conn(0x56357a32a1d0 msgr2=0x56357a3af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 shutdown_connections 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.648+0000 7faaf469ab80 1 -- 172.21.15.98:0/2602010309 wait complete. 2024-03-18T20:26:00.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.649+0000 7faaf469ab80 20 client.15375 trim_cache size 0 max 16384 2024-03-18T20:26:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 Processor -- start 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 -- start start 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3afad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a2ff9c0 con 0x56357a3b9940 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a303880 con 0x56357a3ba7e0 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faaf469ab80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a3b0b00 con 0x56357a3b8fe0 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41520/0 (socket says 172.21.15.98:41520) 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.666+0000 7faae5aa0700 1 -- 172.21.15.98:0/3404134115 learned_addr learned my addr 172.21.15.98:0/3404134115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae62a1700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 -- 172.21.15.98:0/3404134115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 msgr2=0x56357a3c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 -- 172.21.15.98:0/3404134115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 msgr2=0x56357a3afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae6aa2700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 -- 172.21.15.98:0/3404134115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56357a2cce50 con 0x56357a3b9940 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae6aa2700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faae5aa0700 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faad400b9f0 tx=0x7faad4005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9baa09f4024127bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faad17fa700 1 -- 172.21.15.98:0/3404134115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad402b040 con 0x56357a3b9940 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faad17fa700 1 -- 172.21.15.98:0/3404134115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faad4005660 con 0x56357a3b9940 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.667+0000 7faad17fa700 1 -- 172.21.15.98:0/3404134115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4004a60 con 0x56357a3b9940 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- 172.21.15.98:0/3404134115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 msgr2=0x56357a3ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faad400b9f0 tx=0x7faad4005530 comp rx=0 tx=0).stop 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- 172.21.15.98:0/3404134115 shutdown_connections 2024-03-18T20:26:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3ba7e0 0x56357a3afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3b8fe0 0x56357a3c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- 172.21.15.98:0/3404134115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3ca2c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- 172.21.15.98:0/3404134115 >> 172.21.15.98:0/3404134115 conn(0x56357a32a1d0 msgr2=0x56357a3c0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- 172.21.15.98:0/3404134115 shutdown_connections 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- 172.21.15.98:0/3404134115 wait complete. 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- start start 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 Processor -- start 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b8fe0 0x56357a3c4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 0x56357a3c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a3c9fc0 con 0x56357a3b9940 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a3ca130 con 0x56357a3b8fe0 2024-03-18T20:26:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357a3ca2f0 con 0x56357a3ba7e0 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faaf469ab80 10 client.0 _ll_register_callbacks cb 0x56357a10e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae5aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41528/0 (socket says 172.21.15.98:41528) 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae5aa0700 1 -- 172.21.15.98:0/2584767309 learned_addr learned my addr 172.21.15.98:0/2584767309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae62a1700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b8fe0 0x56357a3c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae6aa2700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 0x56357a3c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.668+0000 7faae5aa0700 1 -- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 msgr2=0x56357a3c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae5aa0700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 0x56357a3c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae5aa0700 1 -- 172.21.15.98:0/2584767309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b8fe0 msgr2=0x56357a3c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae5aa0700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b8fe0 0x56357a3c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae5aa0700 1 -- 172.21.15.98:0/2584767309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56357a188360 con 0x56357a3b9940 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae6aa2700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 0x56357a3c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faae5aa0700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faad400baf0 tx=0x7faad4004fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59371490f9738706 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad402b070 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faad401e040 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faad4003d30 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad4003ef0 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 10 client.15381 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56357a18cc80 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faad401e040 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7faad400f460 con 0x56357a3b9940 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 10 client.15381 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 10 client.15381 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 20 client.15381 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a426b60 0x56357a423530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 4 client.15381 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.669+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 --> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56357a155ca0 con 0x56357a426b60 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faae62a1700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a426b60 0x56357a423530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faac9ffb700 10 client.15381 ms_handle_connect on v2:172.21.15.169:6838/2458510608 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faae62a1700 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a426b60 0x56357a423530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34799f5be7b4ddb8 server_cookie=7d468f8751dcb04 in_seq=0 out_seq=0 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faac9ffb700 1 -- 172.21.15.98:0/2584767309 <== mds.1 v2:172.21.15.169:6838/2458510608 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56357a155ca0 con 0x56357a426b60 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faac9ffb700 10 client.15381 handle_command_reply: tid=0 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 client.15381 shutdown 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 msgr2=0x56357a3c4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faad400baf0 tx=0x7faad4004fc0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 shutdown_connections 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:6838/2458510608,v1:172.21.15.169:6839/2458510608] conn(0x56357a426b60 0x56357a423530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56357a3ba7e0 0x56357a3c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56357a3b8fe0 0x56357a3c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 --2- 172.21.15.98:0/2584767309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56357a3b9940 0x56357a3c4aa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faac9ffb700 0 client.15381 ms_handle_reset on v2:172.21.15.169:6838/2458510608 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.670+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 >> 172.21.15.98:0/2584767309 conn(0x56357a32a1d0 msgr2=0x56357a3c08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 shutdown_connections 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faaf469ab80 1 -- 172.21.15.98:0/2584767309 wait complete. 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faaf469ab80 20 client.15381 trim_cache size 0 max 16384 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24475, 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:26:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 62.209249079999999, 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 87, 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 90.198586749 2024-03-18T20:26:00.683 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 msgr2=0x7faacc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 0x7faacc0a3670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faad4000c00 tx=0x7faad4005fb0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 msgr2=0x7faae0196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faadc00d950 tx=0x7faadc00dc60 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 shutdown_connections 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7faacc0a11b0 0x7faacc0a3670 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faae0106a40 0x7faae019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faae01017e0 0x7faae0196be0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 --2- 172.21.15.98:0/3203038172 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faae0104110 0x7faae0197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 >> 172.21.15.98:0/3203038172 conn(0x7faae00fb1d0 msgr2=0x7faae0103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 shutdown_connections 2024-03-18T20:26:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.671+0000 7faae8505700 1 -- 172.21.15.98:0/3203038172 wait complete. 2024-03-18T20:26:00.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 -- 172.21.15.98:0/672455984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c101270 msgr2=0x7f513c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 --2- 172.21.15.98:0/672455984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c101270 0x7f513c101690 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5130009990 tx=0x7f5130009ca0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 -- 172.21.15.98:0/672455984 shutdown_connections 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 --2- 172.21.15.98:0/672455984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c100880 0x7f513c108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 --2- 172.21.15.98:0/672455984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c101270 0x7f513c101690 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 --2- 172.21.15.98:0/672455984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c0ffec0 0x7f513c100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 -- 172.21.15.98:0/672455984 >> 172.21.15.98:0/672455984 conn(0x7f513c0fa950 msgr2=0x7f513c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 -- 172.21.15.98:0/672455984 shutdown_connections 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 -- 172.21.15.98:0/672455984 wait complete. 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.807+0000 7f51416e3700 1 Processor -- start 2024-03-18T20:26:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 -- start start 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c0ffec0 0x7f513c196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c101270 0x7f513c19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c19e750 con 0x7f513c100880 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c19e890 con 0x7f513c0ffec0 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f51416e3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c19ea20 con 0x7f513c101270 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41554/0 (socket says 172.21.15.98:41554) 2024-03-18T20:26:00.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 -- 172.21.15.98:0/2020969945 learned_addr learned my addr 172.21.15.98:0/2020969945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513affd700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c0ffec0 0x7f513c196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513b7fe700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c101270 0x7f513c19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 -- 172.21.15.98:0/2020969945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c101270 msgr2=0x7f513c19e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c101270 0x7f513c19e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 -- 172.21.15.98:0/2020969945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c0ffec0 msgr2=0x7f513c196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c0ffec0 0x7f513c196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 -- 172.21.15.98:0/2020969945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300095f0 con 0x7f513c100880 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.808+0000 7f513a7fc700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c9b0 tx=0x7f512c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.809+0000 7f5123fff700 1 -- 172.21.15.98:0/2020969945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c007b00 con 0x7f513c100880 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.809+0000 7f5123fff700 1 -- 172.21.15.98:0/2020969945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f512c00ce50 con 0x7f513c100880 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.809+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c19ed00 con 0x7f513c100880 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.809+0000 7f5123fff700 1 -- 172.21.15.98:0/2020969945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c018420 con 0x7f513c100880 2024-03-18T20:26:00.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.809+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c19f200 con 0x7f513c100880 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.811+0000 7f5123fff700 1 -- 172.21.15.98:0/2020969945 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f512c00f450 con 0x7f513c100880 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.811+0000 7f5123fff700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 0x7f51240a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.811+0000 7f5123fff700 1 -- 172.21.15.98:0/2020969945 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f512c0ccc30 con 0x7f513c100880 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.811+0000 7f513affd700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 0x7f51240a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f513affd700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 0x7f51240a36f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 Processor -- start 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 -- start start 2024-03-18T20:26:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765409940 con 0x5607654b0bd0 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765407300 con 0x5607654b23d0 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f514d878b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765402bf0 con 0x5607654b1530 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f513affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f513affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41564/0 (socket says 172.21.15.98:41564) 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.812+0000 7f513affd700 1 -- 172.21.15.98:0/2297815086 learned_addr learned my addr 172.21.15.98:0/2297815086 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 -- 172.21.15.98:0/2297815086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 msgr2=0x5607654b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513a7fc700 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 -- 172.21.15.98:0/2297815086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 msgr2=0x5607654b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 -- 172.21.15.98:0/2297815086 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560765284360 con 0x5607654b0bd0 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f513affd700 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5130015000 tx=0x7f5130020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151467df090d7abc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f51217fa700 1 -- 172.21.15.98:0/2297815086 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513002b070 con 0x5607654b0bd0 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f51217fa700 1 -- 172.21.15.98:0/2297815086 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f513001e040 con 0x5607654b0bd0 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.813+0000 7f51217fa700 1 -- 172.21.15.98:0/2297815086 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5130003820 con 0x5607654b0bd0 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- 172.21.15.98:0/2297815086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 msgr2=0x5607654b0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5130015000 tx=0x7f5130020000 comp rx=0 tx=0).stop 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- 172.21.15.98:0/2297815086 shutdown_connections 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b0ff0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- 172.21.15.98:0/2297815086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- 172.21.15.98:0/2297815086 >> 172.21.15.98:0/2297815086 conn(0x5607654261d0 msgr2=0x5607654abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- 172.21.15.98:0/2297815086 shutdown_connections 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- 172.21.15.98:0/2297815086 wait complete. 2024-03-18T20:26:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- start start 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 Processor -- start 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076539ed30 con 0x5607654b23d0 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076539e9d0 con 0x5607654b0bd0 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765286650 con 0x5607654b1530 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f514d878b80 10 client.0 _ll_register_callbacks cb 0x56076520a350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.814+0000 7f513b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41566/0 (socket says 172.21.15.98:41566) 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 -- 172.21.15.98:0/2437011739 learned_addr learned my addr 172.21.15.98:0/2437011739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513affd700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513a7fc700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 -- 172.21.15.98:0/2437011739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 msgr2=0x5607654baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654baf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 -- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 msgr2=0x5607654baa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654baa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 -- 172.21.15.98:0/2437011739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c108a20 con 0x5607654b23d0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f511eff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f513b7fe700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f513400aec0 tx=0x7f513400f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f2646f6f77705c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134019070 con 0x5607654b23d0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5134004bb0 con 0x5607654b23d0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607654a3a80 con 0x5607654b23d0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5134007e80 con 0x5607654b23d0 2024-03-18T20:26:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f514d878b80 10 client.15393 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560765230b10 con 0x5607654b23d0 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.815+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5607654a3a80 con 0x5607654b23d0 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f513401c030 con 0x5607654b23d0 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 10 client.15393 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 10 client.15393 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 20 client.15393 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5607654c5130 0x5607654caf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 4 client.15393 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607653fb6b0 con 0x5607654c5130 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f513affd700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5607654c5130 0x5607654caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f511eff5700 10 client.15393 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.816+0000 7f513affd700 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5607654c5130 0x5607654caf10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e40114dc761b44b server_cookie=1be6d31f2ec283f in_seq=0 out_seq=0 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f511eff5700 1 -- 172.21.15.98:0/2437011739 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5607653fb6b0 con 0x5607654c5130 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f511eff5700 10 client.15393 handle_command_reply: tid=0 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 client.15393 shutdown 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 msgr2=0x5607654c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f513400aec0 tx=0x7f513400f420 comp rx=0 tx=0).stop 2024-03-18T20:26:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 shutdown_connections 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x5607654c5130 0x5607654caf10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c1f90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654baa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 --2- 172.21.15.98:0/2437011739 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f511eff5700 0 client.15393 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 >> 172.21.15.98:0/2437011739 conn(0x5607654261d0 msgr2=0x5607654ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 shutdown_connections 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 1 -- 172.21.15.98:0/2437011739 wait complete. 2024-03-18T20:26:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.817+0000 7f514d878b80 20 client.15393 trim_cache size 0 max 16384 2024-03-18T20:26:00.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 Processor -- start 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 -- start start 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607653ff850 con 0x5607654b0bd0 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607653a1de0 con 0x5607654b23d0 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f514d878b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607654c3110 con 0x5607654b1530 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41578/0 (socket says 172.21.15.98:41578) 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 -- 172.21.15.98:0/1945820745 learned_addr learned my addr 172.21.15.98:0/1945820745 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513b7fe700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513a7fc700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 -- 172.21.15.98:0/1945820745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 msgr2=0x5607654c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 -- 172.21.15.98:0/1945820745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 msgr2=0x5607654b9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 -- 172.21.15.98:0/1945820745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607653c8e50 con 0x5607654b0bd0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513b7fe700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513a7fc700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f513affd700 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f513000bbb0 tx=0x7f513001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8253da7495091467 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f5121ffb700 1 -- 172.21.15.98:0/1945820745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513002b040 con 0x5607654b0bd0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f5121ffb700 1 -- 172.21.15.98:0/1945820745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f513003b0f0 con 0x5607654b0bd0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.835+0000 7f5121ffb700 1 -- 172.21.15.98:0/1945820745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51300047c0 con 0x5607654b0bd0 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- 172.21.15.98:0/1945820745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 msgr2=0x5607654b27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f513000bbb0 tx=0x7f513001a040 comp rx=0 tx=0).stop 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- 172.21.15.98:0/1945820745 shutdown_connections 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b23d0 0x5607654b9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b0bd0 0x5607654b27f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 --2- 172.21.15.98:0/1945820745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- 172.21.15.98:0/1945820745 >> 172.21.15.98:0/1945820745 conn(0x5607654261d0 msgr2=0x5607654b8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- 172.21.15.98:0/1945820745 shutdown_connections 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- 172.21.15.98:0/1945820745 wait complete. 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 -- start start 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.836+0000 7f514d878b80 1 Processor -- start 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654be3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654be920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607654c1a30 con 0x5607654b23d0 2024-03-18T20:26:00.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607654c1ba0 con 0x5607654b0bd0 2024-03-18T20:26:00.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607654c1d60 con 0x5607654b1530 2024-03-18T20:26:00.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 10 client.0 _ll_register_callbacks cb 0x7f513400a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41580/0 (socket says 172.21.15.98:41580) 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 -- 172.21.15.98:0/341571653 learned_addr learned my addr 172.21.15.98:0/341571653 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513affd700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654be3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513a7fc700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654be920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 -- 172.21.15.98:0/341571653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 msgr2=0x5607654be920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654be920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 -- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 msgr2=0x5607654be3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654be3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 -- 172.21.15.98:0/341571653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560765284360 con 0x5607654b23d0 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513a7fc700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654be920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513affd700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654be3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:00.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f513b7fe700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5134007580 tx=0x7f5134014fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8782d98ea129253c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f511eff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134019070 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5134007e80 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607654b9a20 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5134007850 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 10 client.15399 fetch_fsmap learned FSMap version 23 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.837+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560765288c80 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5607654b9a20 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f513402c060 con 0x5607654b23d0 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 10 client.15399 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 10 client.15399 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 20 client.15399 populate_metadata read hostname 'smithi098' 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x56076551ea30 0x56076551b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 4 client.15399 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 --> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f513001e040 con 0x56076551ea30 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f513affd700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x56076551ea30 0x56076551b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f511eff5700 10 client.15399 ms_handle_connect on v2:172.21.15.136:6838/2871006308 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.838+0000 7f513affd700 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x56076551ea30 0x56076551b300 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e1eb76decc0bd88 server_cookie=fbd779db0fa81ec4 in_seq=0 out_seq=0 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f511eff5700 1 -- 172.21.15.98:0/341571653 <== mds.0 v2:172.21.15.136:6838/2871006308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f513001e040 con 0x56076551ea30 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f511eff5700 10 client.15399 handle_command_reply: tid=0 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 client.15399 shutdown 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 msgr2=0x5607654c14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5134007580 tx=0x7f5134014fb0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 shutdown_connections 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:6838/2871006308,v1:172.21.15.136:6839/2871006308] conn(0x56076551ea30 0x56076551b300 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607654b23d0 0x5607654c14f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607654b0bd0 0x5607654be3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f511eff5700 0 client.15399 ms_handle_reset on v2:172.21.15.136:6838/2871006308 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 --2- 172.21.15.98:0/341571653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607654b1530 0x5607654be920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 >> 172.21.15.98:0/341571653 conn(0x5607654261d0 msgr2=0x5607654b8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 shutdown_connections 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 1 -- 172.21.15.98:0/341571653 wait complete. 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f514d878b80 20 client.15399 trim_cache size 0 max 16384 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:26:00.851 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a6e2b072-e564-11ee-95c9-87774f69a715", 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24494, 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 79.500887442999996, 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 87, 2024-03-18T20:26:00.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 85, 2024-03-18T20:26:00.853 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 85.154002359000003 2024-03-18T20:26:00.853 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:26:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 msgr2=0x7f51240a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 0x7f51240a36f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f5130005fb0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 msgr2=0x7f513c197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c9b0 tx=0x7f512c00ccc0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 shutdown_connections 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f513c101270 0x7f513c19e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f51240a1230 0x7f51240a36f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c0ffec0 0x7f513c196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 --2- 172.21.15.98:0/2020969945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f513c100880 0x7f513c197160 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 >> 172.21.15.98:0/2020969945 conn(0x7f513c0fa950 msgr2=0x7f513c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 shutdown_connections 2024-03-18T20:26:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:00.839+0000 7f51416e3700 1 -- 172.21.15.98:0/2020969945 wait complete. 2024-03-18T20:26:00.855 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-18T20:26:00.866 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-18T20:26:00.866 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-18T20:26:00.867 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-18T20:26:00.867 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T20:26:00.867 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:00.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:00.867 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-18T20:26:00.884 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T20:26:00.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:00.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T20:26:00.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:00 smithi098 ceph-mon[43462]: pgmap v264: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:00.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/880104519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:26:00.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2452435698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:26:00.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:00.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-18T20:26:01.065 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-18T20:26:01.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T20:26:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:00 smithi169 ceph-mon[48529]: pgmap v264: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:01.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/880104519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:26:01.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2452435698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:26:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:00 smithi136 ceph-mon[48073]: pgmap v264: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:01.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/880104519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T20:26:01.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2452435698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:26:01.288 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 -- 172.21.15.98:0/4262883876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e480ffc50 msgr2=0x7f4e481000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 --2- 172.21.15.98:0/4262883876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e480ffc50 0x7f4e481000d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009990 tx=0x7f4e38009ca0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 -- 172.21.15.98:0/4262883876 shutdown_connections 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 --2- 172.21.15.98:0/4262883876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e48100610 0x7f4e4810caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 --2- 172.21.15.98:0/4262883876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e481060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 --2- 172.21.15.98:0/4262883876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e480ffc50 0x7f4e481000d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 -- 172.21.15.98:0/4262883876 >> 172.21.15.98:0/4262883876 conn(0x7f4e48076ce0 msgr2=0x7f4e480770f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 -- 172.21.15.98:0/4262883876 shutdown_connections 2024-03-18T20:26:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.218+0000 7f4e4d808700 1 -- 172.21.15.98:0/4262883876 wait complete. 2024-03-18T20:26:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 Processor -- start 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 -- start start 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e480ffc50 0x7f4e4819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e48197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48197970 con 0x7f4e48100610 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48197ae0 con 0x7f4e480ffc50 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e4d808700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e48197ca0 con 0x7f4e48105cf0 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e467fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41608/0 (socket says 172.21.15.98:41608) 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.219+0000 7f4e467fc700 1 -- 172.21.15.98:0/3302246178 learned_addr learned my addr 172.21.15.98:0/3302246178 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e46ffd700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e480ffc50 0x7f4e4819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e477fe700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e48197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 -- 172.21.15.98:0/3302246178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 msgr2=0x7f4e48197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e48197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 -- 172.21.15.98:0/3302246178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e480ffc50 msgr2=0x7f4e4819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e480ffc50 0x7f4e4819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 -- 172.21.15.98:0/3302246178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380095f0 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e477fe700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e48197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e467fc700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4e38004760 tx=0x7f4e38009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3801d070 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e38005030 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e48197ec0 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38004bf0 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.220+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e481a3cf0 con 0x7f4e48100610 2024-03-18T20:26:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.221+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e48101330 con 0x7f4e48100610 2024-03-18T20:26:02.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.222+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f4e38004d50 con 0x7f4e48100610 2024-03-18T20:26:02.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.223+0000 7f4e4c806700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 0x7f4e300a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.223+0000 7f4e46ffd700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 0x7f4e300a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.223+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e380ce380 con 0x7f4e48100610 2024-03-18T20:26:02.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.223+0000 7f4e46ffd700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 0x7f4e300a36f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c005950 tx=0x7f4e3c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.226+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4e380976d0 con 0x7f4e48100610 2024-03-18T20:26:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.465+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4e4804ea80 con 0x7f4e48100610 2024-03-18T20:26:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.466+0000 7f4e4c806700 1 -- 172.21.15.98:0/3302246178 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4e3802f070 con 0x7f4e48100610 2024-03-18T20:26:02.470 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T20:26:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 msgr2=0x7f4e300a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 0x7f4e300a36f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c005950 tx=0x7f4e3c0058e0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 msgr2=0x7f4e4819d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4e38004760 tx=0x7f4e38009570 comp rx=0 tx=0).stop 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 shutdown_connections 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e48105cf0 0x7f4e48197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4e300a1230 0x7f4e300a36f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e480ffc50 0x7f4e4819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 --2- 172.21.15.98:0/3302246178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e48100610 0x7f4e4819d240 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 >> 172.21.15.98:0/3302246178 conn(0x7f4e48076ce0 msgr2=0x7f4e480fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 shutdown_connections 2024-03-18T20:26:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:02.468+0000 7f4e4d808700 1 -- 172.21.15.98:0/3302246178 wait complete. 2024-03-18T20:26:02.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:02 smithi136 ceph-mon[48073]: pgmap v265: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:02.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:02 smithi098 ceph-mon[43462]: pgmap v265: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:02.832 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:26:02.832 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:26:02.833 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:26:02.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:02.833 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:02.852 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41861sec preferred_lft 41861sec 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::6960:dbbe:1345:fedd/64 scope link noprefixroute 2024-03-18T20:26:02.853 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:02.853 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T20:26:02.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T20:26:02.854 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:26:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:02 smithi169 ceph-mon[48529]: pgmap v265: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2 op/s; 58 B/s, 0 keys/s, 1 objects/s recovering 2024-03-18T20:26:02.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:02.974 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:26:03.001 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-18T20:26:03.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.003 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-03-18T20:26:03.063 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-03-18T20:26:03.063 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:26:03.063 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T20:26:03.063 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T20:26:03.063 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:26:03.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T20:26:03.064 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:26:03.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.170 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-18T20:26:03.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.187 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-03-18T20:26:03.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.244 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:26:03.244 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:26:03.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:26:03.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T20:26:03.245 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:26:03.340 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T20:26:03.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T20:26:03.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T20:26:03.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T20:26:03.341 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:26:03.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.535 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2024-03-18T20:26:03.535 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-18T20:26:03.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-03-18T20:26:03.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T20:26:03.535 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-18T20:26:03.633 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-18T20:26:03.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:26:03.634 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:26:03.651 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:26:03.651 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:03.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-18T20:26:03.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:03 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3302246178' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:26:03.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3302246178' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:26:03.776 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:26:03.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:26:03.842 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:26:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:26:03.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:03.845 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-18T20:26:03.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:03 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3302246178' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T20:26:03.902 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:26:03.902 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/1cf87f69-e1b6-4c60-968f-b28ce3aef5d3 --client_fs=cephfs) 2024-03-18T20:26:03.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-18T20:26:03.971 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:26:03.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:26:04.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2024-03-18T20:26:04.030+0000 7f94684025c0 -1 init, newargv = 0x555c12f1bb10 newargc=15 2024-03-18T20:26:04.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[109730]: starting ceph client 2024-03-18T20:26:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:26:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:26:04.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:04.040 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-18T20:26:04.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[109730]: starting fuse 2024-03-18T20:26:04.070 INFO:teuthology.orchestra.run.smithi098.stdout:124 2024-03-18T20:26:04.070 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2024-03-18T20:26:04.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:26:04.070 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:26:04.071 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2024-03-18T20:26:04.072 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-03-18T20:26:04.194 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.109730.asok 2024-03-18T20:26:04.376 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.109730.asok 2024-03-18T20:26:04.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:04.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.109730.asok status 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:26:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "109730", 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/volumes/qa/sv_0/1cf87f69-e1b6-4c60-968f-b28ce3aef5d3" 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 15411, 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 15411 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:26:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3530471039 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3530471039 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.15411 192.168.0.1:0/3530471039", 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/3530471039", 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 5, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 23, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 87, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 85, 2024-03-18T20:26:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-03-18T20:26:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-03-18T20:26:04.539 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:26:04.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:04.546 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-03-18T20:26:04.546 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:04.563 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-03-18T20:26:04.563 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:04.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:04.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:04.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:04 smithi098 ceph-mon[43462]: pgmap v266: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:04.633 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T20:26:04.644 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T20:26:04.644 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:26:04.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-18T20:26:04.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:04 smithi136 ceph-mon[48073]: pgmap v266: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:04.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 -- 172.21.15.98:0/2310118183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c100df0 msgr2=0x7fbc5c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 --2- 172.21.15.98:0/2310118183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c100df0 0x7fbc5c101250 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0099c0 tx=0x7fbc4c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 -- 172.21.15.98:0/2310118183 shutdown_connections 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 --2- 172.21.15.98:0/2310118183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 0x7fbc5c108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 --2- 172.21.15.98:0/2310118183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 --2- 172.21.15.98:0/2310118183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c100df0 0x7fbc5c101250 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 -- 172.21.15.98:0/2310118183 >> 172.21.15.98:0/2310118183 conn(0x7fbc5c0fb170 msgr2=0x7fbc5c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 -- 172.21.15.98:0/2310118183 shutdown_connections 2024-03-18T20:26:04.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.794+0000 7fbc6095a700 1 -- 172.21.15.98:0/2310118183 wait complete. 2024-03-18T20:26:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 Processor -- start 2024-03-18T20:26:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 -- start start 2024-03-18T20:26:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c100df0 0x7fbc5c1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 0x7fbc5c19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c19e6d0 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c19e810 con 0x7fbc5c100df0 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc6095a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c19e9a0 con 0x7fbc5c101790 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44830/0 (socket says 172.21.15.98:44830) 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 -- 172.21.15.98:0/946369840 learned_addr learned my addr 172.21.15.98:0/946369840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc59d9b700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c100df0 0x7fbc5c1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5ad9d700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 0x7fbc5c19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 -- 172.21.15.98:0/946369840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 msgr2=0x7fbc5c19e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 0x7fbc5c19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 -- 172.21.15.98:0/946369840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c100df0 msgr2=0x7fbc5c1970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c100df0 0x7fbc5c1970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 -- 172.21.15.98:0/946369840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0095f0 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc5a59c700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d980 tx=0x7fbc5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.795+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50004020 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.796+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbc50009730 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.796+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c19ec80 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.796+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50003a00 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.796+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c19f190 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.797+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c067c30 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.798+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbc500098a0 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.799+0000 7fbc4b7fe700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 0x7fbc440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.799+0000 7fbc59d9b700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 0x7fbc440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.799+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbc500ccdc0 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.799+0000 7fbc59d9b700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 0x7fbc440a3670 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00b580 tx=0x7fbc4c005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:04.802+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc500960e0 con 0x7fbc5c0ffc00 2024-03-18T20:26:04.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:04 smithi169 ceph-mon[48529]: pgmap v266: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:05.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.044+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fbc5c19f320 con 0x7fbc5c0ffc00 2024-03-18T20:26:05.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.456+0000 7fbc4b7fe700 1 -- 172.21.15.98:0/946369840 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbc50099700 con 0x7fbc5c0ffc00 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 msgr2=0x7fbc440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 0x7fbc440a3670 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00b580 tx=0x7fbc4c005e80 comp rx=0 tx=0).stop 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 msgr2=0x7fbc5c196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d980 tx=0x7fbc5000dc90 comp rx=0 tx=0).stop 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 shutdown_connections 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbc5c101790 0x7fbc5c19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc5c0ffc00 0x7fbc5c196ba0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fbc440a11b0 0x7fbc440a3670 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 --2- 172.21.15.98:0/946369840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc5c100df0 0x7fbc5c1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 >> 172.21.15.98:0/946369840 conn(0x7fbc5c0fb170 msgr2=0x7fbc5c068080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 shutdown_connections 2024-03-18T20:26:05.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.458+0000 7fbc6095a700 1 -- 172.21.15.98:0/946369840 wait complete. 2024-03-18T20:26:05.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 -- 172.21.15.98:0/3879224096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 msgr2=0x7f0690101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 --2- 172.21.15.98:0/3879224096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f0690101250 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f06840099c0 tx=0x7f0684009cd0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 -- 172.21.15.98:0/3879224096 shutdown_connections 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 --2- 172.21.15.98:0/3879224096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f0690108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 --2- 172.21.15.98:0/3879224096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 0x7f0690100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 --2- 172.21.15.98:0/3879224096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f0690101250 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 -- 172.21.15.98:0/3879224096 >> 172.21.15.98:0/3879224096 conn(0x7f06900fb170 msgr2=0x7f06900fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 -- 172.21.15.98:0/3879224096 shutdown_connections 2024-03-18T20:26:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.595+0000 7f0696d15700 1 -- 172.21.15.98:0/3879224096 wait complete. 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 Processor -- start 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 -- start start 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 0x7f0690068ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f069019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069019e730 con 0x7f0690100df0 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069019e8a0 con 0x7f0690101790 2024-03-18T20:26:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0696d15700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069019ea60 con 0x7f06900ffc00 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f068ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f068ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44842/0 (socket says 172.21.15.98:44842) 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f068ffff700 1 -- 172.21.15.98:0/2575410285 learned_addr learned my addr 172.21.15.98:0/2575410285 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f0694ab1700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 0x7f0690068ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f06952b2700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f069019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.596+0000 7f068ffff700 1 -- 172.21.15.98:0/2575410285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 msgr2=0x7f0690068ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068ffff700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 0x7f0690068ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068ffff700 1 -- 172.21.15.98:0/2575410285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 msgr2=0x7f069019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068ffff700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f069019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068ffff700 1 -- 172.21.15.98:0/2575410285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840095f0 con 0x7f0690100df0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f06952b2700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f069019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068ffff700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0684005910 tx=0x7f0684005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068401d070 con 0x7f0690100df0 2024-03-18T20:26:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f068400bd30 con 0x7f0690100df0 2024-03-18T20:26:05.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069019ece0 con 0x7f0690100df0 2024-03-18T20:26:05.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684022d80 con 0x7f0690100df0 2024-03-18T20:26:05.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.597+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069019f100 con 0x7f0690100df0 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.598+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06901070f0 con 0x7f0690100df0 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.600+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f0684022470 con 0x7f0690100df0 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.600+0000 7f068dffb700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 0x7f06780a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.600+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f06840ce010 con 0x7f0690100df0 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.600+0000 7f0694ab1700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 0x7f06780a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.601+0000 7f0694ab1700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 0x7f06780a3670 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0690100c50 tx=0x7f068800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.604+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06840973c0 con 0x7f0690100df0 2024-03-18T20:26:05.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:05 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T20:26:05.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.835+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 --> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f0690061290 con 0x7f06780a11b0 2024-03-18T20:26:05.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:05 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T20:26:05.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.840+0000 7f068dffb700 1 -- 172.21.15.98:0/2575410285 <== mgr.14152 v2:172.21.15.98:6800/3114713517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f0690061290 con 0x7f06780a11b0 2024-03-18T20:26:05.843 INFO:teuthology.orchestra.run.smithi098.stdout:{} 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 msgr2=0x7f06780a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 0x7f06780a3670 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0690100c50 tx=0x7f068800d040 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 msgr2=0x7f069019b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0684005910 tx=0x7f0684005b00 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 shutdown_connections 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06780a11b0 0x7f06780a3670 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0690101790 0x7f069019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06900ffc00 0x7f0690068ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 --2- 172.21.15.98:0/2575410285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0690100df0 0x7f069019b9a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 >> 172.21.15.98:0/2575410285 conn(0x7f06900fb170 msgr2=0x7f0690104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 shutdown_connections 2024-03-18T20:26:05.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:26:05.842+0000 7f0696d15700 1 -- 172.21.15.98:0/2575410285 wait complete. 2024-03-18T20:26:05.858 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T20:26:05.870 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T20:26:05.889 INFO:tasks.workunit:Pulling workunits from ref 49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:26:05.889 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T20:26:05.889 INFO:tasks.workunit:timeout=3h 2024-03-18T20:26:05.889 INFO:tasks.workunit:cleanup=True 2024-03-18T20:26:05.890 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:05.890 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:05 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 7ch/124d Inode: 1 Links: 2 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-18 20:25:01.272939655 +0000 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-18 20:25:01.272939655 +0000 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-18 20:26:04.627658897 +0000 2024-03-18T20:26:05.909 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-03-18T20:26:05.909 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:05.910 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T20:26:05.983 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T20:26:06.040 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T20:26:06.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:06 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T20:26:06.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:06 smithi136 ceph-mon[48073]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:26:06.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:06 smithi136 ceph-mon[48073]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:06.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:06 smithi136 ceph-mon[48073]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:26:06.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:06 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T20:26:06.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:06 smithi098 ceph-mon[43462]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:26:06.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:06 smithi098 ceph-mon[43462]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:06.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:06 smithi098 ceph-mon[43462]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:26:06.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:06 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/946369840' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T20:26:06.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:06 smithi169 ceph-mon[48529]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T20:26:06.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:06 smithi169 ceph-mon[48529]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:06.884 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:06 smithi169 ceph-mon[48529]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T20:26:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:08 smithi136 ceph-mon[48073]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-18T20:26:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:08 smithi098 ceph-mon[43462]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-18T20:26:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:08 smithi169 ceph-mon[48529]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-18T20:26:10.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:10 smithi136 ceph-mon[48073]: pgmap v269: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:26:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:10 smithi098 ceph-mon[43462]: pgmap v269: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:26:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:10 smithi169 ceph-mon[48529]: pgmap v269: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:26:12.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:12 smithi136 ceph-mon[48073]: pgmap v270: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:12 smithi098 ceph-mon[43462]: pgmap v270: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:12 smithi169 ceph-mon[48529]: pgmap v270: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:14 smithi136 ceph-mon[48073]: pgmap v271: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:14 smithi098 ceph-mon[43462]: pgmap v271: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:14 smithi169 ceph-mon[48529]: pgmap v271: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-18T20:26:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:16 smithi136 ceph-mon[48073]: pgmap v272: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-18T20:26:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:16 smithi098 ceph-mon[43462]: pgmap v272: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-18T20:26:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:16 smithi169 ceph-mon[48529]: pgmap v272: 129 pgs: 129 active+clean; 652 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-18T20:26:18.993 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:18 smithi136 ceph-mon[48073]: pgmap v273: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T20:26:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:18 smithi098 ceph-mon[43462]: pgmap v273: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T20:26:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:18 smithi169 ceph-mon[48529]: pgmap v273: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T20:26:20.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:20 smithi136 ceph-mon[48073]: pgmap v274: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-18T20:26:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:20 smithi098 ceph-mon[43462]: pgmap v274: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-18T20:26:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:20 smithi169 ceph-mon[48529]: pgmap v274: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-18T20:26:22.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:22 smithi136 ceph-mon[48073]: pgmap v275: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T20:26:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:22 smithi098 ceph-mon[43462]: pgmap v275: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T20:26:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:22 smithi169 ceph-mon[48529]: pgmap v275: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-18T20:26:24.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:24 smithi136 ceph-mon[48073]: pgmap v276: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:26:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:24 smithi098 ceph-mon[43462]: pgmap v276: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:26:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:24 smithi169 ceph-mon[48529]: pgmap v276: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:26:26.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:26 smithi136 ceph-mon[48073]: pgmap v277: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T20:26:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:26 smithi098 ceph-mon[43462]: pgmap v277: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T20:26:27.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:26 smithi169 ceph-mon[48529]: pgmap v277: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-18T20:26:28.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:28 smithi136 ceph-mon[48073]: pgmap v278: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-18T20:26:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:28 smithi098 ceph-mon[43462]: pgmap v278: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-18T20:26:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:28 smithi169 ceph-mon[48529]: pgmap v278: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-18T20:26:30.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:30 smithi136 ceph-mon[48073]: pgmap v279: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:30 smithi098 ceph-mon[43462]: pgmap v279: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:31.137 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:30 smithi169 ceph-mon[48529]: pgmap v279: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-18T20:26:32.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:32 smithi136 ceph-mon[48073]: pgmap v280: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-18T20:26:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:32 smithi098 ceph-mon[43462]: pgmap v280: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-18T20:26:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:32 smithi169 ceph-mon[48529]: pgmap v280: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-18T20:26:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:34 smithi136 ceph-mon[48073]: pgmap v281: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:34 smithi098 ceph-mon[43462]: pgmap v281: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:34 smithi169 ceph-mon[48529]: pgmap v281: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:36.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:36 smithi136 ceph-mon[48073]: pgmap v282: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:36 smithi098 ceph-mon[43462]: pgmap v282: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:36 smithi169 ceph-mon[48529]: pgmap v282: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:38 smithi136 ceph-mon[48073]: pgmap v283: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:38 smithi098 ceph-mon[43462]: pgmap v283: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:38 smithi169 ceph-mon[48529]: pgmap v283: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:40.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:40 smithi136 ceph-mon[48073]: pgmap v284: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:40 smithi098 ceph-mon[43462]: pgmap v284: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:40 smithi169 ceph-mon[48529]: pgmap v284: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:41.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:41 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:26:42.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:41 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:26:42.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:41 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:26:43.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: pgmap v285: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T20:26:43.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:26:43.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:42 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: pgmap v285: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T20:26:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:26:43.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:42 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: pgmap v285: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:26:43.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:42 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:26:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:44 smithi098 ceph-mon[43462]: pgmap v286: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:44 smithi169 ceph-mon[48529]: pgmap v286: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:44 smithi136 ceph-mon[48073]: pgmap v286: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T20:26:46.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-18T20:26:46.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-03-18T20:26:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-18T20:26:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T20:26:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T20:26:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-18T20:26:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-18T20:26:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-03-18T20:26:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-18T20:26:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T20:26:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T20:26:46.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-18T20:26:46.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-18T20:26:46.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]: dispatch 2024-03-18T20:26:46.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-18T20:26:46.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T20:26:46.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T20:26:46.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-18T20:26:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: pgmap v287: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-18T20:26:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:46 smithi098 ceph-mon[43462]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T20:26:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: pgmap v287: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-18T20:26:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-03-18T20:26:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-18T20:26:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T20:26:47.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T20:26:47.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-18T20:26:47.135 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:46 smithi169 ceph-mon[48529]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T20:26:47.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: pgmap v287: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3b", "id": [3, 9]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-18T20:26:47.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:46 smithi136 ceph-mon[48073]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T20:26:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:47 smithi098 ceph-mon[43462]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T20:26:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:47 smithi169 ceph-mon[48529]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T20:26:48.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:47 smithi136 ceph-mon[48073]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T20:26:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:48 smithi098 ceph-mon[43462]: pgmap v290: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:48 smithi098 ceph-mon[43462]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-18T20:26:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:48 smithi169 ceph-mon[48529]: pgmap v290: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:49.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:48 smithi169 ceph-mon[48529]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-18T20:26:49.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:48 smithi136 ceph-mon[48073]: pgmap v290: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:49.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:48 smithi136 ceph-mon[48073]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-18T20:26:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:50 smithi098 ceph-mon[43462]: pgmap v291: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:26:51.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:50 smithi169 ceph-mon[48529]: pgmap v291: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:26:51.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:50 smithi136 ceph-mon[48073]: pgmap v291: 129 pgs: 2 activating, 4 peering, 123 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T20:26:52.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:52 smithi136 ceph-mon[48073]: pgmap v292: 129 pgs: 2 activating, 127 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T20:26:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:52 smithi098 ceph-mon[43462]: pgmap v292: 129 pgs: 2 activating, 127 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T20:26:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:52 smithi169 ceph-mon[48529]: pgmap v292: 129 pgs: 2 activating, 127 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T20:26:53.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:53 smithi136 ceph-mon[48073]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-18T20:26:53.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:53 smithi136 ceph-mon[48073]: Cluster is now healthy 2024-03-18T20:26:53.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:53 smithi098 ceph-mon[43462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-18T20:26:53.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:53 smithi098 ceph-mon[43462]: Cluster is now healthy 2024-03-18T20:26:53.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:53 smithi169 ceph-mon[48529]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-18T20:26:53.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:53 smithi169 ceph-mon[48529]: Cluster is now healthy 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '49bdc51142759622a281d3e5327eebd07cbb8bb3'. 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T20:26:54.570 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:54.571 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 49bdc511427 Merge branch 'wip-63770-reef' of https://github.com/idryomov/ceph into wip-yuri10-testing-2024-03-15-1653-reef 2024-03-18T20:26:54.575 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T20:26:54.637 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T20:26:54.639 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T20:26:54.639 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T20:26:54.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:54 smithi136 ceph-mon[48073]: pgmap v293: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:54.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:54 smithi098 ceph-mon[43462]: pgmap v293: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:54.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:54 smithi169 ceph-mon[48529]: pgmap v293: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-18T20:26:55.118 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T20:26:55.176 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T20:26:55.213 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T20:26:55.214 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T20:26:55.214 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T20:26:55.251 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T20:26:55.254 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:26:55.255 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T20:26:55.313 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-18T20:26:55.314 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-18T20:26:55.314 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49bdc51142759622a281d3e5327eebd07cbb8bb3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-18T20:26:55.380 INFO:tasks.workunit.client.0.smithi098.stdout:getting blogbench 2024-03-18T20:26:55.403 INFO:tasks.workunit.client.0.smithi098.stderr:--2024-03-18 20:26:55-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-18T20:26:55.404 INFO:tasks.workunit.client.0.smithi098.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-18T20:26:55.441 INFO:tasks.workunit.client.0.smithi098.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-18T20:26:55.477 INFO:tasks.workunit.client.0.smithi098.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-18T20:26:55.478 INFO:tasks.workunit.client.0.smithi098.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-18T20:26:55.480 INFO:tasks.workunit.client.0.smithi098.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-18T20:26:55.482 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:55.558 INFO:tasks.workunit.client.0.smithi098.stderr: 0K .......... .......... .......... .......... .......... 49% 646K 0s 2024-03-18T20:26:55.616 INFO:tasks.workunit.client.0.smithi098.stderr: 50K .......... .......... .......... .......... .......... 99% 860K 0s 2024-03-18T20:26:55.617 INFO:tasks.workunit.client.0.smithi098.stderr: 100K 100% 1779G=0.1s 2024-03-18T20:26:55.617 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:55.617 INFO:tasks.workunit.client.0.smithi098.stderr:2024-03-18 20:26:55 (744 KB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-18T20:26:55.617 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-18T20:26:55.653 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/ 2024-03-18T20:26:55.655 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/m4/ 2024-03-18T20:26:55.656 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-18T20:26:55.659 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/man/ 2024-03-18T20:26:55.661 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/man/blogbench.8 2024-03-18T20:26:55.664 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/man/Makefile.am 2024-03-18T20:26:55.666 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/man/Makefile.in 2024-03-18T20:26:55.668 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/ 2024-03-18T20:26:55.670 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-18T20:26:55.673 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-18T20:26:55.675 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/blogbench.c 2024-03-18T20:26:55.677 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/blogbench.h 2024-03-18T20:26:55.679 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-18T20:26:55.681 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-18T20:26:55.683 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-18T20:26:55.685 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-18T20:26:55.688 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-18T20:26:55.690 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/Makefile.am 2024-03-18T20:26:55.693 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/Makefile.in 2024-03-18T20:26:55.696 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/process.c 2024-03-18T20:26:55.698 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/globals.h 2024-03-18T20:26:55.700 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/commenter.c 2024-03-18T20:26:55.703 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/rewriter.c 2024-03-18T20:26:55.705 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/reader.c 2024-03-18T20:26:55.707 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/writer.c 2024-03-18T20:26:55.709 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/src/helpers.c 2024-03-18T20:26:55.711 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/NEWS 2024-03-18T20:26:55.714 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/compile 2024-03-18T20:26:55.716 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/depcomp 2024-03-18T20:26:55.718 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/aclocal.m4 2024-03-18T20:26:55.720 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/README 2024-03-18T20:26:55.722 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/configure 2024-03-18T20:26:55.730 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/configure.ac 2024-03-18T20:26:55.732 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/config.guess 2024-03-18T20:26:55.735 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/install-sh 2024-03-18T20:26:55.737 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/config.sub 2024-03-18T20:26:55.740 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/missing 2024-03-18T20:26:55.742 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/mkinstalldirs 2024-03-18T20:26:55.744 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/Makefile.am 2024-03-18T20:26:55.746 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/Makefile.in 2024-03-18T20:26:55.748 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/config.h.in 2024-03-18T20:26:55.751 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/AUTHORS 2024-03-18T20:26:55.752 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/INSTALL 2024-03-18T20:26:55.755 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/ChangeLog 2024-03-18T20:26:55.757 INFO:tasks.workunit.client.0.smithi098.stdout:blogbench-1.0/COPYING 2024-03-18T20:26:55.761 INFO:tasks.workunit.client.0.smithi098.stdout:making blogbench 2024-03-18T20:26:55.866 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T20:26:56.878 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether build environment is sane... yes 2024-03-18T20:26:56.897 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gawk... gawk 2024-03-18T20:26:56.922 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T20:26:56.938 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc... gcc 2024-03-18T20:26:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:56 smithi136 ceph-mon[48073]: pgmap v294: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-18T20:26:57.015 INFO:tasks.workunit.client.0.smithi098.stdout:checking for C compiler default output file name... a.out 2024-03-18T20:26:57.017 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether the C compiler works... yes 2024-03-18T20:26:57.021 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are cross compiling... no 2024-03-18T20:26:57.070 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of executables... 2024-03-18T20:26:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:56 smithi098 ceph-mon[43462]: pgmap v294: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-18T20:26:57.116 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of object files... o 2024-03-18T20:26:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:56 smithi169 ceph-mon[48529]: pgmap v294: 129 pgs: 129 active+clean; 659 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-18T20:26:57.165 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T20:26:57.211 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc accepts -g... yes 2024-03-18T20:26:57.266 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-18T20:26:57.329 INFO:tasks.workunit.client.0.smithi098.stdout:checking for style of include used by make... GNU 2024-03-18T20:26:57.482 INFO:tasks.workunit.client.0.smithi098.stdout:checking dependency style of gcc... gcc3 2024-03-18T20:26:57.483 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T20:26:57.541 INFO:tasks.workunit.client.0.smithi098.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T20:26:57.597 INFO:tasks.workunit.client.0.smithi098.stdout:checking for egrep... grep -E 2024-03-18T20:26:57.610 INFO:tasks.workunit.client.0.smithi098.stdout:checking for AIX... no 2024-03-18T20:26:57.701 INFO:tasks.workunit.client.0.smithi098.stdout:checking for library containing strerror... none required 2024-03-18T20:26:57.878 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ANSI C header files... yes 2024-03-18T20:26:57.933 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T20:26:57.994 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/types.h... yes 2024-03-18T20:26:58.063 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/stat.h... yes 2024-03-18T20:26:58.169 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdlib.h... yes 2024-03-18T20:26:58.237 INFO:tasks.workunit.client.0.smithi098.stdout:checking for string.h... yes 2024-03-18T20:26:58.306 INFO:tasks.workunit.client.0.smithi098.stdout:checking for memory.h... yes 2024-03-18T20:26:58.382 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strings.h... yes 2024-03-18T20:26:58.455 INFO:tasks.workunit.client.0.smithi098.stdout:checking for inttypes.h... yes 2024-03-18T20:26:58.522 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdint.h... yes 2024-03-18T20:26:58.593 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... yes 2024-03-18T20:26:58.611 INFO:tasks.workunit.client.0.smithi098.stdout:checking for memory.h... (cached) yes 2024-03-18T20:26:58.623 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strings.h... (cached) yes 2024-03-18T20:26:58.634 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... (cached) yes 2024-03-18T20:26:58.708 INFO:tasks.workunit.client.0.smithi098.stdout:checking locale.h usability... yes 2024-03-18T20:26:58.735 INFO:tasks.workunit.client.0.smithi098.stdout:checking locale.h presence... yes 2024-03-18T20:26:58.736 INFO:tasks.workunit.client.0.smithi098.stdout:checking for locale.h... yes 2024-03-18T20:26:58.759 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdlib.h... (cached) yes 2024-03-18T20:26:58.770 INFO:tasks.workunit.client.0.smithi098.stdout:checking for string.h... (cached) yes 2024-03-18T20:26:58.837 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/socket.h usability... yes 2024-03-18T20:26:58.864 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/socket.h presence... yes 2024-03-18T20:26:58.865 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/socket.h... yes 2024-03-18T20:26:58.944 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h usability... yes 2024-03-18T20:26:58.970 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h presence... yes 2024-03-18T20:26:58.971 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/time.h... yes 2024-03-18T20:26:58.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:26:58 smithi136 ceph-mon[48073]: pgmap v295: 129 pgs: 129 active+clean; 953 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-03-18T20:26:59.045 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h usability... yes 2024-03-18T20:26:59.073 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h presence... yes 2024-03-18T20:26:59.074 INFO:tasks.workunit.client.0.smithi098.stdout:checking for fcntl.h... yes 2024-03-18T20:26:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:26:58 smithi098 ceph-mon[43462]: pgmap v295: 129 pgs: 129 active+clean; 953 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-03-18T20:26:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:26:58 smithi169 ceph-mon[48529]: pgmap v295: 129 pgs: 129 active+clean; 953 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 7 op/s 2024-03-18T20:26:59.157 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/fcntl.h usability... yes 2024-03-18T20:26:59.182 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/fcntl.h presence... yes 2024-03-18T20:26:59.183 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/fcntl.h... yes 2024-03-18T20:26:59.271 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/param.h usability... yes 2024-03-18T20:26:59.300 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/param.h presence... yes 2024-03-18T20:26:59.301 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/param.h... yes 2024-03-18T20:26:59.371 INFO:tasks.workunit.client.0.smithi098.stdout:checking ioctl.h usability... no 2024-03-18T20:26:59.402 INFO:tasks.workunit.client.0.smithi098.stdout:checking ioctl.h presence... no 2024-03-18T20:26:59.403 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ioctl.h... no 2024-03-18T20:26:59.495 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/ioctl.h usability... yes 2024-03-18T20:26:59.522 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/ioctl.h presence... yes 2024-03-18T20:26:59.523 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/ioctl.h... yes 2024-03-18T20:26:59.590 INFO:tasks.workunit.client.0.smithi098.stdout:checking return type of signal handlers... void 2024-03-18T20:26:59.649 INFO:tasks.workunit.client.0.smithi098.stdout:checking for size_t... yes 2024-03-18T20:26:59.700 INFO:tasks.workunit.client.0.smithi098.stdout:checking for dev_t... yes 2024-03-18T20:26:59.755 INFO:tasks.workunit.client.0.smithi098.stdout:checking for short... yes 2024-03-18T20:26:59.853 INFO:tasks.workunit.client.0.smithi098.stdout:checking size of short... 2 2024-03-18T20:26:59.917 INFO:tasks.workunit.client.0.smithi098.stdout:checking for int... yes 2024-03-18T20:27:00.005 INFO:tasks.workunit.client.0.smithi098.stdout:checking size of int... 4 2024-03-18T20:27:00.063 INFO:tasks.workunit.client.0.smithi098.stdout:checking for long... yes 2024-03-18T20:27:00.155 INFO:tasks.workunit.client.0.smithi098.stdout:checking size of long... 8 2024-03-18T20:27:00.215 INFO:tasks.workunit.client.0.smithi098.stdout:checking for long long... yes 2024-03-18T20:27:00.305 INFO:tasks.workunit.client.0.smithi098.stdout:checking size of long long... 8 2024-03-18T20:27:00.362 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ssize_t... yes 2024-03-18T20:27:00.420 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sig_atomic_t... yes 2024-03-18T20:27:00.489 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc needs -traditional... no 2024-03-18T20:27:00.536 INFO:tasks.workunit.client.0.smithi098.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T20:27:00.571 INFO:tasks.workunit.client.0.smithi098.stdout:checking for inline... inline 2024-03-18T20:27:00.656 INFO:tasks.workunit.client.0.smithi098.stdout:checking for working alloca.h... yes 2024-03-18T20:27:00.726 INFO:tasks.workunit.client.0.smithi098.stdout:checking for alloca... yes 2024-03-18T20:27:00.808 INFO:tasks.workunit.client.0.smithi098.stdout:checking for working memcmp... yes 2024-03-18T20:27:00.888 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strftime... yes 2024-03-18T20:27:00.984 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-18T20:27:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:00 smithi136 ceph-mon[48073]: pgmap v296: 129 pgs: 129 active+clean; 1.4 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 65 KiB/s wr, 11 op/s 2024-03-18T20:27:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:00 smithi098 ceph-mon[43462]: pgmap v296: 129 pgs: 129 active+clean; 1.4 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 65 KiB/s wr, 11 op/s 2024-03-18T20:27:01.093 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether stat accepts an empty string... no 2024-03-18T20:27:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:00 smithi169 ceph-mon[48529]: pgmap v296: 129 pgs: 129 active+clean; 1.4 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 65 KiB/s wr, 11 op/s 2024-03-18T20:27:01.179 INFO:tasks.workunit.client.0.smithi098.stdout:checking for vprintf... yes 2024-03-18T20:27:01.268 INFO:tasks.workunit.client.0.smithi098.stdout:checking for _doprnt... no 2024-03-18T20:27:01.300 INFO:tasks.workunit.client.0.smithi098.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T20:27:01.316 INFO:tasks.workunit.client.0.smithi098.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T20:27:01.382 INFO:tasks.workunit.client.0.smithi098.stdout:checking for the pthreads library -lpthreads... no 2024-03-18T20:27:01.491 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether pthreads work without any flags... no 2024-03-18T20:27:01.518 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether pthreads work with -Kthread... no 2024-03-18T20:27:01.547 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether pthreads work with -kthread... no 2024-03-18T20:27:01.605 INFO:tasks.workunit.client.0.smithi098.stdout:checking for the pthreads library -llthread... no 2024-03-18T20:27:01.680 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether pthreads work with -pthread... yes 2024-03-18T20:27:01.746 INFO:tasks.workunit.client.0.smithi098.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-18T20:27:01.747 INFO:tasks.workunit.client.0.smithi098.stdout:checking if more special flags are required for pthreads... no 2024-03-18T20:27:01.747 INFO:tasks.workunit.client.0.smithi098.stdout:checking for cc_r... gcc 2024-03-18T20:27:01.831 INFO:tasks.workunit.client.0.smithi098.stdout:checking for setrlimit... yes 2024-03-18T20:27:01.924 INFO:tasks.workunit.client.0.smithi098.stdout:checking for setproctitle... no 2024-03-18T20:27:02.010 INFO:tasks.workunit.client.0.smithi098.stdout:checking for getopt_long... yes 2024-03-18T20:27:02.098 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strtoull... yes 2024-03-18T20:27:02.183 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strtoq... yes 2024-03-18T20:27:02.272 INFO:tasks.workunit.client.0.smithi098.stdout:checking for snprintf... yes 2024-03-18T20:27:02.361 INFO:tasks.workunit.client.0.smithi098.stdout:checking for vsnprintf... yes 2024-03-18T20:27:02.467 INFO:tasks.workunit.client.0.smithi098.stdout:checking for vfprintf... yes 2024-03-18T20:27:02.559 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gethostname... yes 2024-03-18T20:27:02.644 INFO:tasks.workunit.client.0.smithi098.stdout:checking for setlocale... yes 2024-03-18T20:27:02.729 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sync... yes 2024-03-18T20:27:02.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:02 smithi136 ceph-mon[48073]: pgmap v297: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 307 KiB/s wr, 45 op/s 2024-03-18T20:27:02.817 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether snprintf is C99 conformant... done 2024-03-18T20:27:02.836 INFO:tasks.workunit.client.0.smithi098.stdout:configure: creating ./config.status 2024-03-18T20:27:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:02 smithi098 ceph-mon[43462]: pgmap v297: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 307 KiB/s wr, 45 op/s 2024-03-18T20:27:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:02 smithi169 ceph-mon[48529]: pgmap v297: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 307 KiB/s wr, 45 op/s 2024-03-18T20:27:03.082 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating Makefile 2024-03-18T20:27:03.104 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating src/Makefile 2024-03-18T20:27:03.130 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating man/Makefile 2024-03-18T20:27:03.147 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating config.h 2024-03-18T20:27:03.218 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: executing depfiles commands 2024-03-18T20:27:03.406 INFO:tasks.workunit.client.0.smithi098.stdout:make all-recursive 2024-03-18T20:27:03.413 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-18T20:27:03.417 INFO:tasks.workunit.client.0.smithi098.stdout:Making all in src 2024-03-18T20:27:03.461 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-18T20:27:03.461 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-18T20:27:03.461 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-18T20:27:03.544 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-18T20:27:03.544 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-18T20:27:03.637 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-18T20:27:03.638 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-18T20:27:03.717 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-18T20:27:03.717 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-18T20:27:03.796 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-18T20:27:03.796 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-18T20:27:03.873 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-18T20:27:03.873 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-18T20:27:03.947 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-18T20:27:03.948 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-18T20:27:04.048 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-18T20:27:04.048 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-18T20:27:04.111 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-18T20:27:04.111 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-18T20:27:04.173 INFO:tasks.workunit.client.0.smithi098.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-18T20:27:04.173 INFO:tasks.workunit.client.0.smithi098.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-18T20:27:04.229 INFO:tasks.workunit.client.0.smithi098.stdout:gcc -g -O2 -pthread -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-18T20:27:04.286 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-18T20:27:04.287 INFO:tasks.workunit.client.0.smithi098.stdout:Making all in man 2024-03-18T20:27:04.301 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-18T20:27:04.301 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Nothing to be done for 'all'. 2024-03-18T20:27:04.301 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-18T20:27:04.310 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-18T20:27:04.312 INFO:tasks.workunit.client.0.smithi098.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-18T20:27:04.313 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-18T20:27:04.318 INFO:tasks.workunit.client.0.smithi098.stdout:running blogbench 2024-03-18T20:27:04.319 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-03-18T20:27:04.319 INFO:tasks.workunit.client.0.smithi098.stdout:Frequency = 10 secs 2024-03-18T20:27:04.319 INFO:tasks.workunit.client.0.smithi098.stdout:Scratch dir = [blogtest_in] 2024-03-18T20:27:04.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:04 smithi136 ceph-mon[48073]: pgmap v298: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 332 KiB/s wr, 65 op/s 2024-03-18T20:27:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:04 smithi098 ceph-mon[43462]: pgmap v298: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 332 KiB/s wr, 65 op/s 2024-03-18T20:27:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:04 smithi169 ceph-mon[48529]: pgmap v298: 129 pgs: 129 active+clean; 3.9 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 332 KiB/s wr, 65 op/s 2024-03-18T20:27:06.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:06 smithi136 ceph-mon[48073]: pgmap v299: 129 pgs: 129 active+clean; 4.2 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 411 KiB/s wr, 104 op/s 2024-03-18T20:27:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:06 smithi098 ceph-mon[43462]: pgmap v299: 129 pgs: 129 active+clean; 4.2 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 411 KiB/s wr, 104 op/s 2024-03-18T20:27:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:06 smithi169 ceph-mon[48529]: pgmap v299: 129 pgs: 129 active+clean; 4.2 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 411 KiB/s wr, 104 op/s 2024-03-18T20:27:07.574 INFO:tasks.workunit.client.0.smithi098.stdout:Spawning 3 writers... 2024-03-18T20:27:07.575 INFO:tasks.workunit.client.0.smithi098.stdout:Spawning 1 rewriters... 2024-03-18T20:27:07.575 INFO:tasks.workunit.client.0.smithi098.stdout:Spawning 5 commenters... 2024-03-18T20:27:07.575 INFO:tasks.workunit.client.0.smithi098.stdout:Spawning 100 readers... 2024-03-18T20:27:07.577 INFO:tasks.workunit.client.0.smithi098.stdout:Benchmarking for 30 iterations. 2024-03-18T20:27:07.577 INFO:tasks.workunit.client.0.smithi098.stdout:The test will run during 5 minutes. 2024-03-18T20:27:07.577 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-03-18T20:27:07.577 INFO:tasks.workunit.client.0.smithi098.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-18T20:27:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:08 smithi136 ceph-mon[48073]: pgmap v300: 129 pgs: 129 active+clean; 5.8 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 566 KiB/s wr, 122 op/s 2024-03-18T20:27:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:08 smithi098 ceph-mon[43462]: pgmap v300: 129 pgs: 129 active+clean; 5.8 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 566 KiB/s wr, 122 op/s 2024-03-18T20:27:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:08 smithi169 ceph-mon[48529]: pgmap v300: 129 pgs: 129 active+clean; 5.8 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 566 KiB/s wr, 122 op/s 2024-03-18T20:27:10.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:10 smithi136 ceph-mon[48073]: pgmap v301: 129 pgs: 129 active+clean; 6.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 601 KiB/s wr, 127 op/s 2024-03-18T20:27:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:10 smithi098 ceph-mon[43462]: pgmap v301: 129 pgs: 129 active+clean; 6.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 601 KiB/s wr, 127 op/s 2024-03-18T20:27:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:10 smithi169 ceph-mon[48529]: pgmap v301: 129 pgs: 129 active+clean; 6.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 601 KiB/s wr, 127 op/s 2024-03-18T20:27:12.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:12 smithi136 ceph-mon[48073]: pgmap v302: 129 pgs: 129 active+clean; 15 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-03-18T20:27:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:12 smithi098 ceph-mon[43462]: pgmap v302: 129 pgs: 129 active+clean; 15 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-03-18T20:27:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:12 smithi169 ceph-mon[48529]: pgmap v302: 129 pgs: 129 active+clean; 15 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-03-18T20:27:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:14 smithi136 ceph-mon[48073]: pgmap v303: 129 pgs: 129 active+clean; 20 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-03-18T20:27:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:14 smithi098 ceph-mon[43462]: pgmap v303: 129 pgs: 129 active+clean; 20 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-03-18T20:27:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:14 smithi169 ceph-mon[48529]: pgmap v303: 129 pgs: 129 active+clean; 20 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-03-18T20:27:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:16 smithi136 ceph-mon[48073]: pgmap v304: 129 pgs: 129 active+clean; 26 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 2.1 MiB/s wr, 164 op/s 2024-03-18T20:27:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:16 smithi098 ceph-mon[43462]: pgmap v304: 129 pgs: 129 active+clean; 26 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 2.1 MiB/s wr, 164 op/s 2024-03-18T20:27:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:16 smithi169 ceph-mon[48529]: pgmap v304: 129 pgs: 129 active+clean; 26 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 2.1 MiB/s wr, 164 op/s 2024-03-18T20:27:17.577 INFO:tasks.workunit.client.0.smithi098.stdout: 6 12478 418 4543 376 3646 460 2024-03-18T20:27:17.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:17 smithi136 ceph-mon[48073]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-18T20:27:18.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:17 smithi098 ceph-mon[43462]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-18T20:27:18.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:17 smithi169 ceph-mon[48529]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-18T20:27:18.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:18 smithi136 ceph-mon[48073]: pgmap v305: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-18T20:27:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:18 smithi098 ceph-mon[43462]: pgmap v305: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-18T20:27:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:18 smithi169 ceph-mon[48529]: pgmap v305: 129 pgs: 129 active+clean; 31 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-18T20:27:20.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:20 smithi136 ceph-mon[48073]: pgmap v306: 129 pgs: 129 active+clean; 34 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-03-18T20:27:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:20 smithi098 ceph-mon[43462]: pgmap v306: 129 pgs: 129 active+clean; 34 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-03-18T20:27:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:20 smithi169 ceph-mon[48529]: pgmap v306: 129 pgs: 129 active+clean; 34 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 2.6 MiB/s wr, 138 op/s 2024-03-18T20:27:22.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:22 smithi136 ceph-mon[48073]: pgmap v307: 129 pgs: 129 active+clean; 38 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-18T20:27:22.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:22 smithi098 ceph-mon[43462]: pgmap v307: 129 pgs: 129 active+clean; 38 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-18T20:27:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:22 smithi169 ceph-mon[48529]: pgmap v307: 129 pgs: 129 active+clean; 38 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-18T20:27:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:24 smithi136 ceph-mon[48073]: pgmap v308: 129 pgs: 129 active+clean; 39 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.2 MiB/s wr, 132 op/s 2024-03-18T20:27:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:24 smithi098 ceph-mon[43462]: pgmap v308: 129 pgs: 129 active+clean; 39 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.2 MiB/s wr, 132 op/s 2024-03-18T20:27:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:24 smithi169 ceph-mon[48529]: pgmap v308: 129 pgs: 129 active+clean; 39 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.2 MiB/s wr, 132 op/s 2024-03-18T20:27:26.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:26 smithi136 ceph-mon[48073]: pgmap v309: 129 pgs: 129 active+clean; 44 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-18T20:27:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:26 smithi098 ceph-mon[43462]: pgmap v309: 129 pgs: 129 active+clean; 44 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-18T20:27:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:26 smithi169 ceph-mon[48529]: pgmap v309: 129 pgs: 129 active+clean; 44 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-18T20:27:27.577 INFO:tasks.workunit.client.0.smithi098.stdout: 8 14808 251 7708 146 3502 228 2024-03-18T20:27:28.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:28 smithi136 ceph-mon[48073]: pgmap v310: 129 pgs: 129 active+clean; 45 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-18T20:27:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:28 smithi098 ceph-mon[43462]: pgmap v310: 129 pgs: 129 active+clean; 45 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-18T20:27:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:28 smithi169 ceph-mon[48529]: pgmap v310: 129 pgs: 129 active+clean; 45 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-18T20:27:30.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:30 smithi136 ceph-mon[48073]: pgmap v311: 129 pgs: 129 active+clean; 48 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-03-18T20:27:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:30 smithi098 ceph-mon[43462]: pgmap v311: 129 pgs: 129 active+clean; 48 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-03-18T20:27:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:30 smithi169 ceph-mon[48529]: pgmap v311: 129 pgs: 129 active+clean; 48 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-03-18T20:27:32.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:32 smithi136 ceph-mon[48073]: pgmap v312: 129 pgs: 129 active+clean; 53 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-18T20:27:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:32 smithi098 ceph-mon[43462]: pgmap v312: 129 pgs: 129 active+clean; 53 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-18T20:27:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:32 smithi169 ceph-mon[48529]: pgmap v312: 129 pgs: 129 active+clean; 53 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-18T20:27:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:34 smithi136 ceph-mon[48073]: pgmap v313: 129 pgs: 129 active+clean; 54 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-03-18T20:27:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:34 smithi098 ceph-mon[43462]: pgmap v313: 129 pgs: 129 active+clean; 54 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-03-18T20:27:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:34 smithi169 ceph-mon[48529]: pgmap v313: 129 pgs: 129 active+clean; 54 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.6 MiB/s wr, 96 op/s 2024-03-18T20:27:36.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:36 smithi136 ceph-mon[48073]: pgmap v314: 129 pgs: 129 active+clean; 56 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-18T20:27:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:36 smithi098 ceph-mon[43462]: pgmap v314: 129 pgs: 129 active+clean; 56 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-18T20:27:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:36 smithi169 ceph-mon[48529]: pgmap v314: 129 pgs: 129 active+clean; 56 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-18T20:27:37.577 INFO:tasks.workunit.client.0.smithi098.stdout: 11 14360 170 8240 133 3464 221 2024-03-18T20:27:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:38 smithi136 ceph-mon[48073]: pgmap v315: 129 pgs: 129 active+clean; 58 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 92 op/s 2024-03-18T20:27:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:38 smithi098 ceph-mon[43462]: pgmap v315: 129 pgs: 129 active+clean; 58 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 92 op/s 2024-03-18T20:27:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:38 smithi169 ceph-mon[48529]: pgmap v315: 129 pgs: 129 active+clean; 58 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 92 op/s 2024-03-18T20:27:40.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:40 smithi136 ceph-mon[48073]: pgmap v316: 129 pgs: 129 active+clean; 60 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-03-18T20:27:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:40 smithi098 ceph-mon[43462]: pgmap v316: 129 pgs: 129 active+clean; 60 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-03-18T20:27:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:40 smithi169 ceph-mon[48529]: pgmap v316: 129 pgs: 129 active+clean; 60 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-03-18T20:27:42.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:42 smithi136 ceph-mon[48073]: pgmap v317: 129 pgs: 129 active+clean; 63 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-18T20:27:42.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:42 smithi098 ceph-mon[43462]: pgmap v317: 129 pgs: 129 active+clean; 63 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-18T20:27:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:42 smithi169 ceph-mon[48529]: pgmap v317: 129 pgs: 129 active+clean; 63 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-18T20:27:43.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:27:43.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:27:43.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:27:43.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:27:43.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:27:43.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:27:43.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:27:43.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:27:43.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:27:43.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:27:43.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:27:43.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:27:44.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:44 smithi136 ceph-mon[48073]: pgmap v318: 129 pgs: 129 active+clean; 64 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-03-18T20:27:44.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:44 smithi098 ceph-mon[43462]: pgmap v318: 129 pgs: 129 active+clean; 64 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-03-18T20:27:44.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:44 smithi169 ceph-mon[48529]: pgmap v318: 129 pgs: 129 active+clean; 64 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-03-18T20:27:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:46 smithi136 ceph-mon[48073]: pgmap v319: 129 pgs: 129 active+clean; 67 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 1.3 MiB/s wr, 89 op/s 2024-03-18T20:27:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:46 smithi098 ceph-mon[43462]: pgmap v319: 129 pgs: 129 active+clean; 67 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 1.3 MiB/s wr, 89 op/s 2024-03-18T20:27:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:46 smithi169 ceph-mon[48529]: pgmap v319: 129 pgs: 129 active+clean; 67 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 1.3 MiB/s wr, 89 op/s 2024-03-18T20:27:47.577 INFO:tasks.workunit.client.0.smithi098.stdout: 12 14257 118 8575 105 3532 240 2024-03-18T20:27:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:48 smithi136 ceph-mon[48073]: pgmap v320: 129 pgs: 129 active+clean; 68 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:27:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:48 smithi098 ceph-mon[43462]: pgmap v320: 129 pgs: 129 active+clean; 68 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:27:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:48 smithi169 ceph-mon[48529]: pgmap v320: 129 pgs: 129 active+clean; 68 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:27:50.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:50 smithi136 ceph-mon[48073]: pgmap v321: 129 pgs: 129 active+clean; 70 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.1 MiB/s wr, 77 op/s 2024-03-18T20:27:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:50 smithi098 ceph-mon[43462]: pgmap v321: 129 pgs: 129 active+clean; 70 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.1 MiB/s wr, 77 op/s 2024-03-18T20:27:51.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:50 smithi169 ceph-mon[48529]: pgmap v321: 129 pgs: 129 active+clean; 70 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.1 MiB/s wr, 77 op/s 2024-03-18T20:27:52.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:52 smithi136 ceph-mon[48073]: pgmap v322: 129 pgs: 129 active+clean; 73 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-03-18T20:27:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:52 smithi098 ceph-mon[43462]: pgmap v322: 129 pgs: 129 active+clean; 73 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-03-18T20:27:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:52 smithi169 ceph-mon[48529]: pgmap v322: 129 pgs: 129 active+clean; 73 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 87 op/s 2024-03-18T20:27:54.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:54 smithi136 ceph-mon[48073]: pgmap v323: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:27:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:54 smithi098 ceph-mon[43462]: pgmap v323: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:27:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:54 smithi169 ceph-mon[48529]: pgmap v323: 129 pgs: 129 active+clean; 74 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:27:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:56 smithi136 ceph-mon[48073]: pgmap v324: 129 pgs: 129 active+clean; 76 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-18T20:27:57.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:56 smithi098 ceph-mon[43462]: pgmap v324: 129 pgs: 129 active+clean; 76 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-18T20:27:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:56 smithi169 ceph-mon[48529]: pgmap v324: 129 pgs: 129 active+clean; 76 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-18T20:27:57.577 INFO:tasks.workunit.client.0.smithi098.stdout: 14 14315 87 8685 102 4128 258 2024-03-18T20:27:58.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:27:58 smithi136 ceph-mon[48073]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-03-18T20:27:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:27:58 smithi098 ceph-mon[43462]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-03-18T20:27:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:27:58 smithi169 ceph-mon[48529]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 81 op/s 2024-03-18T20:28:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:00 smithi136 ceph-mon[48073]: pgmap v326: 129 pgs: 129 active+clean; 79 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-03-18T20:28:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:00 smithi098 ceph-mon[43462]: pgmap v326: 129 pgs: 129 active+clean; 79 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-03-18T20:28:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:00 smithi169 ceph-mon[48529]: pgmap v326: 129 pgs: 129 active+clean; 79 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-03-18T20:28:02.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:02 smithi136 ceph-mon[48073]: pgmap v327: 129 pgs: 129 active+clean; 82 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-03-18T20:28:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:02 smithi098 ceph-mon[43462]: pgmap v327: 129 pgs: 129 active+clean; 82 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-03-18T20:28:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:02 smithi169 ceph-mon[48529]: pgmap v327: 129 pgs: 129 active+clean; 82 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-03-18T20:28:04.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:04 smithi136 ceph-mon[48073]: pgmap v328: 129 pgs: 129 active+clean; 82 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 958 KiB/s wr, 78 op/s 2024-03-18T20:28:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:04 smithi098 ceph-mon[43462]: pgmap v328: 129 pgs: 129 active+clean; 82 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 958 KiB/s wr, 78 op/s 2024-03-18T20:28:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:04 smithi169 ceph-mon[48529]: pgmap v328: 129 pgs: 129 active+clean; 82 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 958 KiB/s wr, 78 op/s 2024-03-18T20:28:06.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:06 smithi136 ceph-mon[48073]: pgmap v329: 129 pgs: 129 active+clean; 83 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 970 KiB/s wr, 81 op/s 2024-03-18T20:28:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:06 smithi098 ceph-mon[43462]: pgmap v329: 129 pgs: 129 active+clean; 83 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 970 KiB/s wr, 81 op/s 2024-03-18T20:28:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:06 smithi169 ceph-mon[48529]: pgmap v329: 129 pgs: 129 active+clean; 83 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 970 KiB/s wr, 81 op/s 2024-03-18T20:28:07.577 INFO:tasks.workunit.client.0.smithi098.stdout: 15 14414 147 9206 38 3788 185 2024-03-18T20:28:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:08 smithi136 ceph-mon[48073]: pgmap v330: 129 pgs: 129 active+clean; 85 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 984 KiB/s wr, 76 op/s 2024-03-18T20:28:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:08 smithi098 ceph-mon[43462]: pgmap v330: 129 pgs: 129 active+clean; 85 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 984 KiB/s wr, 76 op/s 2024-03-18T20:28:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:08 smithi169 ceph-mon[48529]: pgmap v330: 129 pgs: 129 active+clean; 85 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 984 KiB/s wr, 76 op/s 2024-03-18T20:28:10.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:10 smithi136 ceph-mon[48073]: pgmap v331: 129 pgs: 129 active+clean; 86 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 854 KiB/s wr, 72 op/s 2024-03-18T20:28:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:10 smithi098 ceph-mon[43462]: pgmap v331: 129 pgs: 129 active+clean; 86 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 854 KiB/s wr, 72 op/s 2024-03-18T20:28:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:10 smithi169 ceph-mon[48529]: pgmap v331: 129 pgs: 129 active+clean; 86 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 854 KiB/s wr, 72 op/s 2024-03-18T20:28:12.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:12 smithi136 ceph-mon[48073]: pgmap v332: 129 pgs: 129 active+clean; 88 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 941 KiB/s wr, 76 op/s 2024-03-18T20:28:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:12 smithi098 ceph-mon[43462]: pgmap v332: 129 pgs: 129 active+clean; 88 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 941 KiB/s wr, 76 op/s 2024-03-18T20:28:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:12 smithi169 ceph-mon[48529]: pgmap v332: 129 pgs: 129 active+clean; 88 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 941 KiB/s wr, 76 op/s 2024-03-18T20:28:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:14 smithi136 ceph-mon[48073]: pgmap v333: 129 pgs: 129 active+clean; 88 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 719 KiB/s wr, 67 op/s 2024-03-18T20:28:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:14 smithi098 ceph-mon[43462]: pgmap v333: 129 pgs: 129 active+clean; 88 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 719 KiB/s wr, 67 op/s 2024-03-18T20:28:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:14 smithi169 ceph-mon[48529]: pgmap v333: 129 pgs: 129 active+clean; 88 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 719 KiB/s wr, 67 op/s 2024-03-18T20:28:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:16 smithi136 ceph-mon[48073]: pgmap v334: 129 pgs: 129 active+clean; 90 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 835 KiB/s wr, 68 op/s 2024-03-18T20:28:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:16 smithi098 ceph-mon[43462]: pgmap v334: 129 pgs: 129 active+clean; 90 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 835 KiB/s wr, 68 op/s 2024-03-18T20:28:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:16 smithi169 ceph-mon[48529]: pgmap v334: 129 pgs: 129 active+clean; 90 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 835 KiB/s wr, 68 op/s 2024-03-18T20:28:17.577 INFO:tasks.workunit.client.0.smithi098.stdout: 15 13843 102 8453 55 5205 122 2024-03-18T20:28:18.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:18 smithi136 ceph-mon[48073]: pgmap v335: 129 pgs: 129 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 743 KiB/s wr, 60 op/s 2024-03-18T20:28:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:18 smithi098 ceph-mon[43462]: pgmap v335: 129 pgs: 129 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 743 KiB/s wr, 60 op/s 2024-03-18T20:28:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:18 smithi169 ceph-mon[48529]: pgmap v335: 129 pgs: 129 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 743 KiB/s wr, 60 op/s 2024-03-18T20:28:20.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:20 smithi136 ceph-mon[48073]: pgmap v336: 129 pgs: 129 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 664 KiB/s wr, 56 op/s 2024-03-18T20:28:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:20 smithi098 ceph-mon[43462]: pgmap v336: 129 pgs: 129 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 664 KiB/s wr, 56 op/s 2024-03-18T20:28:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:20 smithi169 ceph-mon[48529]: pgmap v336: 129 pgs: 129 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 664 KiB/s wr, 56 op/s 2024-03-18T20:28:22.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:22 smithi136 ceph-mon[48073]: pgmap v337: 129 pgs: 129 active+clean; 94 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 855 KiB/s wr, 66 op/s 2024-03-18T20:28:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:22 smithi098 ceph-mon[43462]: pgmap v337: 129 pgs: 129 active+clean; 94 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 855 KiB/s wr, 66 op/s 2024-03-18T20:28:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:22 smithi169 ceph-mon[48529]: pgmap v337: 129 pgs: 129 active+clean; 94 MiB data, 649 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 855 KiB/s wr, 66 op/s 2024-03-18T20:28:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:24 smithi136 ceph-mon[48073]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 746 KiB/s wr, 62 op/s 2024-03-18T20:28:25.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:24 smithi098 ceph-mon[43462]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 746 KiB/s wr, 62 op/s 2024-03-18T20:28:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:24 smithi169 ceph-mon[48529]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 746 KiB/s wr, 62 op/s 2024-03-18T20:28:26.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:26 smithi136 ceph-mon[48073]: pgmap v339: 129 pgs: 129 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1022 KiB/s wr, 74 op/s 2024-03-18T20:28:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:26 smithi098 ceph-mon[43462]: pgmap v339: 129 pgs: 129 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1022 KiB/s wr, 74 op/s 2024-03-18T20:28:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:26 smithi169 ceph-mon[48529]: pgmap v339: 129 pgs: 129 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1022 KiB/s wr, 74 op/s 2024-03-18T20:28:27.577 INFO:tasks.workunit.client.0.smithi098.stdout: 16 13409 35 7922 118 4949 178 2024-03-18T20:28:28.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:28 smithi136 ceph-mon[48073]: pgmap v340: 129 pgs: 129 active+clean; 99 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 871 KiB/s wr, 68 op/s 2024-03-18T20:28:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:28 smithi098 ceph-mon[43462]: pgmap v340: 129 pgs: 129 active+clean; 99 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 871 KiB/s wr, 68 op/s 2024-03-18T20:28:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:28 smithi169 ceph-mon[48529]: pgmap v340: 129 pgs: 129 active+clean; 99 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 871 KiB/s wr, 68 op/s 2024-03-18T20:28:30.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:30 smithi136 ceph-mon[48073]: pgmap v341: 129 pgs: 129 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-03-18T20:28:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:30 smithi098 ceph-mon[43462]: pgmap v341: 129 pgs: 129 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-03-18T20:28:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:30 smithi169 ceph-mon[48529]: pgmap v341: 129 pgs: 129 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-03-18T20:28:32.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:32 smithi136 ceph-mon[48073]: pgmap v342: 129 pgs: 129 active+clean; 102 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:28:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:32 smithi098 ceph-mon[43462]: pgmap v342: 129 pgs: 129 active+clean; 102 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:28:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:32 smithi169 ceph-mon[48529]: pgmap v342: 129 pgs: 129 active+clean; 102 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 80 op/s 2024-03-18T20:28:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:34 smithi136 ceph-mon[48073]: pgmap v343: 129 pgs: 129 active+clean; 103 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 935 KiB/s wr, 72 op/s 2024-03-18T20:28:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:34 smithi098 ceph-mon[43462]: pgmap v343: 129 pgs: 129 active+clean; 103 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 935 KiB/s wr, 72 op/s 2024-03-18T20:28:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:34 smithi169 ceph-mon[48529]: pgmap v343: 129 pgs: 129 active+clean; 103 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 935 KiB/s wr, 72 op/s 2024-03-18T20:28:36.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:36 smithi136 ceph-mon[48073]: pgmap v344: 129 pgs: 129 active+clean; 107 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.2 MiB/s wr, 79 op/s 2024-03-18T20:28:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:36 smithi098 ceph-mon[43462]: pgmap v344: 129 pgs: 129 active+clean; 107 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.2 MiB/s wr, 79 op/s 2024-03-18T20:28:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:36 smithi169 ceph-mon[48529]: pgmap v344: 129 pgs: 129 active+clean; 107 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.2 MiB/s wr, 79 op/s 2024-03-18T20:28:37.577 INFO:tasks.workunit.client.0.smithi098.stdout: 17 13652 69 8759 109 3920 178 2024-03-18T20:28:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:38 smithi136 ceph-mon[48073]: pgmap v345: 129 pgs: 129 active+clean; 107 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 938 KiB/s wr, 66 op/s 2024-03-18T20:28:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:38 smithi098 ceph-mon[43462]: pgmap v345: 129 pgs: 129 active+clean; 107 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 938 KiB/s wr, 66 op/s 2024-03-18T20:28:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:38 smithi169 ceph-mon[48529]: pgmap v345: 129 pgs: 129 active+clean; 107 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 938 KiB/s wr, 66 op/s 2024-03-18T20:28:40.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:40 smithi136 ceph-mon[48073]: pgmap v346: 129 pgs: 129 active+clean; 108 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 978 KiB/s wr, 68 op/s 2024-03-18T20:28:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:40 smithi098 ceph-mon[43462]: pgmap v346: 129 pgs: 129 active+clean; 108 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 978 KiB/s wr, 68 op/s 2024-03-18T20:28:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:40 smithi169 ceph-mon[48529]: pgmap v346: 129 pgs: 129 active+clean; 108 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 978 KiB/s wr, 68 op/s 2024-03-18T20:28:42.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:42 smithi136 ceph-mon[48073]: pgmap v347: 129 pgs: 129 active+clean; 111 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:28:42.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:42 smithi098 ceph-mon[43462]: pgmap v347: 129 pgs: 129 active+clean; 111 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:28:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:42 smithi169 ceph-mon[48529]: pgmap v347: 129 pgs: 129 active+clean; 111 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:28:43.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:43 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:28:43.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:43 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:28:44.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:43 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:28:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:44 smithi136 ceph-mon[48073]: pgmap v348: 129 pgs: 129 active+clean; 112 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1010 KiB/s wr, 63 op/s 2024-03-18T20:28:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:28:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:28:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:28:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:44 smithi098 ceph-mon[43462]: pgmap v348: 129 pgs: 129 active+clean; 112 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1010 KiB/s wr, 63 op/s 2024-03-18T20:28:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:28:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:28:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:28:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:44 smithi169 ceph-mon[48529]: pgmap v348: 129 pgs: 129 active+clean; 112 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1010 KiB/s wr, 63 op/s 2024-03-18T20:28:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:28:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:28:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:28:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:46 smithi136 ceph-mon[48073]: pgmap v349: 129 pgs: 129 active+clean; 115 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-03-18T20:28:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:46 smithi098 ceph-mon[43462]: pgmap v349: 129 pgs: 129 active+clean; 115 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-03-18T20:28:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:46 smithi169 ceph-mon[48529]: pgmap v349: 129 pgs: 129 active+clean; 115 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-03-18T20:28:47.577 INFO:tasks.workunit.client.0.smithi098.stdout: 20 13751 87 9040 93 4241 175 2024-03-18T20:28:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:48 smithi136 ceph-mon[48073]: pgmap v350: 129 pgs: 129 active+clean; 116 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 856 KiB/s wr, 53 op/s 2024-03-18T20:28:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:48 smithi098 ceph-mon[43462]: pgmap v350: 129 pgs: 129 active+clean; 116 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 856 KiB/s wr, 53 op/s 2024-03-18T20:28:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:48 smithi169 ceph-mon[48529]: pgmap v350: 129 pgs: 129 active+clean; 116 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 856 KiB/s wr, 53 op/s 2024-03-18T20:28:50.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:50 smithi136 ceph-mon[48073]: pgmap v351: 129 pgs: 129 active+clean; 117 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 916 KiB/s wr, 55 op/s 2024-03-18T20:28:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:50 smithi098 ceph-mon[43462]: pgmap v351: 129 pgs: 129 active+clean; 117 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 916 KiB/s wr, 55 op/s 2024-03-18T20:28:51.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:50 smithi169 ceph-mon[48529]: pgmap v351: 129 pgs: 129 active+clean; 117 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 916 KiB/s wr, 55 op/s 2024-03-18T20:28:52.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:52 smithi136 ceph-mon[48073]: pgmap v352: 129 pgs: 129 active+clean; 121 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 62 op/s 2024-03-18T20:28:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:52 smithi098 ceph-mon[43462]: pgmap v352: 129 pgs: 129 active+clean; 121 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 62 op/s 2024-03-18T20:28:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:52 smithi169 ceph-mon[48529]: pgmap v352: 129 pgs: 129 active+clean; 121 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 1.1 MiB/s wr, 62 op/s 2024-03-18T20:28:54.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:54 smithi136 ceph-mon[48073]: pgmap v353: 129 pgs: 129 active+clean; 121 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 947 KiB/s wr, 60 op/s 2024-03-18T20:28:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:54 smithi098 ceph-mon[43462]: pgmap v353: 129 pgs: 129 active+clean; 121 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 947 KiB/s wr, 60 op/s 2024-03-18T20:28:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:54 smithi169 ceph-mon[48529]: pgmap v353: 129 pgs: 129 active+clean; 121 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 947 KiB/s wr, 60 op/s 2024-03-18T20:28:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:56 smithi136 ceph-mon[48073]: pgmap v354: 129 pgs: 129 active+clean; 124 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.1 MiB/s wr, 66 op/s 2024-03-18T20:28:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:56 smithi098 ceph-mon[43462]: pgmap v354: 129 pgs: 129 active+clean; 124 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.1 MiB/s wr, 66 op/s 2024-03-18T20:28:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:56 smithi169 ceph-mon[48529]: pgmap v354: 129 pgs: 129 active+clean; 124 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.1 MiB/s wr, 66 op/s 2024-03-18T20:28:57.578 INFO:tasks.workunit.client.0.smithi098.stdout: 21 13961 112 9563 63 3763 169 2024-03-18T20:28:58.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:28:58 smithi136 ceph-mon[48073]: pgmap v355: 129 pgs: 129 active+clean; 124 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 863 KiB/s wr, 61 op/s 2024-03-18T20:28:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:28:58 smithi098 ceph-mon[43462]: pgmap v355: 129 pgs: 129 active+clean; 124 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 863 KiB/s wr, 61 op/s 2024-03-18T20:28:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:28:58 smithi169 ceph-mon[48529]: pgmap v355: 129 pgs: 129 active+clean; 124 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 863 KiB/s wr, 61 op/s 2024-03-18T20:29:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:00 smithi136 ceph-mon[48073]: pgmap v356: 129 pgs: 129 active+clean; 127 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-03-18T20:29:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:00 smithi098 ceph-mon[43462]: pgmap v356: 129 pgs: 129 active+clean; 127 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-03-18T20:29:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:00 smithi169 ceph-mon[48529]: pgmap v356: 129 pgs: 129 active+clean; 127 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-03-18T20:29:02.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:02 smithi136 ceph-mon[48073]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:29:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:02 smithi098 ceph-mon[43462]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:29:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:02 smithi169 ceph-mon[48529]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:29:04.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:04 smithi136 ceph-mon[48073]: pgmap v358: 129 pgs: 129 active+clean; 128 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 781 KiB/s wr, 57 op/s 2024-03-18T20:29:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:04 smithi098 ceph-mon[43462]: pgmap v358: 129 pgs: 129 active+clean; 128 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 781 KiB/s wr, 57 op/s 2024-03-18T20:29:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:04 smithi169 ceph-mon[48529]: pgmap v358: 129 pgs: 129 active+clean; 128 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 781 KiB/s wr, 57 op/s 2024-03-18T20:29:06.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:06 smithi136 ceph-mon[48073]: pgmap v359: 129 pgs: 129 active+clean; 132 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-18T20:29:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:06 smithi098 ceph-mon[43462]: pgmap v359: 129 pgs: 129 active+clean; 132 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-18T20:29:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:06 smithi169 ceph-mon[48529]: pgmap v359: 129 pgs: 129 active+clean; 132 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-18T20:29:07.578 INFO:tasks.workunit.client.0.smithi098.stdout: 21 13691 77 8940 54 4534 137 2024-03-18T20:29:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:08 smithi136 ceph-mon[48073]: pgmap v360: 129 pgs: 129 active+clean; 132 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 850 KiB/s wr, 54 op/s 2024-03-18T20:29:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:08 smithi098 ceph-mon[43462]: pgmap v360: 129 pgs: 129 active+clean; 132 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 850 KiB/s wr, 54 op/s 2024-03-18T20:29:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:08 smithi169 ceph-mon[48529]: pgmap v360: 129 pgs: 129 active+clean; 132 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 850 KiB/s wr, 54 op/s 2024-03-18T20:29:10.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:10 smithi136 ceph-mon[48073]: pgmap v361: 129 pgs: 129 active+clean; 133 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 884 KiB/s wr, 55 op/s 2024-03-18T20:29:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:10 smithi098 ceph-mon[43462]: pgmap v361: 129 pgs: 129 active+clean; 133 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 884 KiB/s wr, 55 op/s 2024-03-18T20:29:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:10 smithi169 ceph-mon[48529]: pgmap v361: 129 pgs: 129 active+clean; 133 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 884 KiB/s wr, 55 op/s 2024-03-18T20:29:12.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:12 smithi136 ceph-mon[48073]: pgmap v362: 129 pgs: 129 active+clean; 136 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 886 KiB/s wr, 60 op/s 2024-03-18T20:29:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:12 smithi098 ceph-mon[43462]: pgmap v362: 129 pgs: 129 active+clean; 136 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 886 KiB/s wr, 60 op/s 2024-03-18T20:29:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:12 smithi169 ceph-mon[48529]: pgmap v362: 129 pgs: 129 active+clean; 136 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 886 KiB/s wr, 60 op/s 2024-03-18T20:29:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:14 smithi136 ceph-mon[48073]: pgmap v363: 129 pgs: 129 active+clean; 136 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 59 op/s 2024-03-18T20:29:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:14 smithi098 ceph-mon[43462]: pgmap v363: 129 pgs: 129 active+clean; 136 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 59 op/s 2024-03-18T20:29:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:14 smithi169 ceph-mon[48529]: pgmap v363: 129 pgs: 129 active+clean; 136 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 831 KiB/s wr, 59 op/s 2024-03-18T20:29:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:16 smithi136 ceph-mon[48073]: pgmap v364: 129 pgs: 129 active+clean; 139 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:29:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:16 smithi098 ceph-mon[43462]: pgmap v364: 129 pgs: 129 active+clean; 139 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:29:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:16 smithi169 ceph-mon[48529]: pgmap v364: 129 pgs: 129 active+clean; 139 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:29:17.578 INFO:tasks.workunit.client.0.smithi098.stdout: 21 13229 48 8096 69 5286 144 2024-03-18T20:29:18.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:18 smithi136 ceph-mon[48073]: pgmap v365: 129 pgs: 129 active+clean; 140 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 769 KiB/s wr, 57 op/s 2024-03-18T20:29:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:18 smithi098 ceph-mon[43462]: pgmap v365: 129 pgs: 129 active+clean; 140 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 769 KiB/s wr, 57 op/s 2024-03-18T20:29:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:18 smithi169 ceph-mon[48529]: pgmap v365: 129 pgs: 129 active+clean; 140 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 769 KiB/s wr, 57 op/s 2024-03-18T20:29:20.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:20 smithi136 ceph-mon[48073]: pgmap v366: 129 pgs: 129 active+clean; 140 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 814 KiB/s wr, 58 op/s 2024-03-18T20:29:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:20 smithi098 ceph-mon[43462]: pgmap v366: 129 pgs: 129 active+clean; 140 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 814 KiB/s wr, 58 op/s 2024-03-18T20:29:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:20 smithi169 ceph-mon[48529]: pgmap v366: 129 pgs: 129 active+clean; 140 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 814 KiB/s wr, 58 op/s 2024-03-18T20:29:22.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:22 smithi136 ceph-mon[48073]: pgmap v367: 129 pgs: 129 active+clean; 144 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-03-18T20:29:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:22 smithi098 ceph-mon[43462]: pgmap v367: 129 pgs: 129 active+clean; 144 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-03-18T20:29:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:22 smithi169 ceph-mon[48529]: pgmap v367: 129 pgs: 129 active+clean; 144 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.0 MiB/s wr, 72 op/s 2024-03-18T20:29:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:24 smithi136 ceph-mon[48073]: pgmap v368: 129 pgs: 129 active+clean; 144 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 857 KiB/s wr, 66 op/s 2024-03-18T20:29:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:24 smithi098 ceph-mon[43462]: pgmap v368: 129 pgs: 129 active+clean; 144 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 857 KiB/s wr, 66 op/s 2024-03-18T20:29:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:24 smithi169 ceph-mon[48529]: pgmap v368: 129 pgs: 129 active+clean; 144 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 857 KiB/s wr, 66 op/s 2024-03-18T20:29:26.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:26 smithi136 ceph-mon[48073]: pgmap v369: 129 pgs: 129 active+clean; 147 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:29:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:26 smithi098 ceph-mon[43462]: pgmap v369: 129 pgs: 129 active+clean; 147 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:29:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:26 smithi169 ceph-mon[48529]: pgmap v369: 129 pgs: 129 active+clean; 147 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1.1 MiB/s wr, 76 op/s 2024-03-18T20:29:27.578 INFO:tasks.workunit.client.0.smithi098.stdout: 22 13637 40 7938 60 5313 128 2024-03-18T20:29:28.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:28 smithi136 ceph-mon[48073]: pgmap v370: 129 pgs: 129 active+clean; 147 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-03-18T20:29:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:28 smithi098 ceph-mon[43462]: pgmap v370: 129 pgs: 129 active+clean; 147 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-03-18T20:29:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:28 smithi169 ceph-mon[48529]: pgmap v370: 129 pgs: 129 active+clean; 147 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-03-18T20:29:30.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:30 smithi136 ceph-mon[48073]: pgmap v371: 129 pgs: 129 active+clean; 148 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 838 KiB/s wr, 67 op/s 2024-03-18T20:29:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:30 smithi098 ceph-mon[43462]: pgmap v371: 129 pgs: 129 active+clean; 148 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 838 KiB/s wr, 67 op/s 2024-03-18T20:29:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:30 smithi169 ceph-mon[48529]: pgmap v371: 129 pgs: 129 active+clean; 148 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 838 KiB/s wr, 67 op/s 2024-03-18T20:29:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:32 smithi098 ceph-mon[43462]: pgmap v372: 129 pgs: 129 active+clean; 150 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-03-18T20:29:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:32 smithi169 ceph-mon[48529]: pgmap v372: 129 pgs: 129 active+clean; 150 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-03-18T20:29:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:32 smithi136 ceph-mon[48073]: pgmap v372: 129 pgs: 129 active+clean; 150 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 74 op/s 2024-03-18T20:29:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:34 smithi136 ceph-mon[48073]: pgmap v373: 129 pgs: 129 active+clean; 151 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 761 KiB/s wr, 61 op/s 2024-03-18T20:29:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:34 smithi098 ceph-mon[43462]: pgmap v373: 129 pgs: 129 active+clean; 151 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 761 KiB/s wr, 61 op/s 2024-03-18T20:29:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:34 smithi169 ceph-mon[48529]: pgmap v373: 129 pgs: 129 active+clean; 151 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 761 KiB/s wr, 61 op/s 2024-03-18T20:29:36.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:36 smithi136 ceph-mon[48073]: pgmap v374: 129 pgs: 129 active+clean; 154 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 991 KiB/s wr, 67 op/s 2024-03-18T20:29:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:36 smithi098 ceph-mon[43462]: pgmap v374: 129 pgs: 129 active+clean; 154 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 991 KiB/s wr, 67 op/s 2024-03-18T20:29:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:36 smithi169 ceph-mon[48529]: pgmap v374: 129 pgs: 129 active+clean; 154 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 991 KiB/s wr, 67 op/s 2024-03-18T20:29:37.578 INFO:tasks.workunit.client.0.smithi098.stdout: 23 13402 73 8664 31 5183 131 2024-03-18T20:29:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:38 smithi136 ceph-mon[48073]: pgmap v375: 129 pgs: 129 active+clean; 154 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 734 KiB/s wr, 55 op/s 2024-03-18T20:29:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:38 smithi098 ceph-mon[43462]: pgmap v375: 129 pgs: 129 active+clean; 154 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 734 KiB/s wr, 55 op/s 2024-03-18T20:29:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:38 smithi169 ceph-mon[48529]: pgmap v375: 129 pgs: 129 active+clean; 154 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 734 KiB/s wr, 55 op/s 2024-03-18T20:29:40.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:40 smithi136 ceph-mon[48073]: pgmap v376: 129 pgs: 129 active+clean; 155 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 748 KiB/s wr, 56 op/s 2024-03-18T20:29:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:40 smithi098 ceph-mon[43462]: pgmap v376: 129 pgs: 129 active+clean; 155 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 748 KiB/s wr, 56 op/s 2024-03-18T20:29:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:40 smithi169 ceph-mon[48529]: pgmap v376: 129 pgs: 129 active+clean; 155 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 748 KiB/s wr, 56 op/s 2024-03-18T20:29:42.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:42 smithi136 ceph-mon[48073]: pgmap v377: 129 pgs: 129 active+clean; 158 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 982 KiB/s wr, 64 op/s 2024-03-18T20:29:42.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:42 smithi098 ceph-mon[43462]: pgmap v377: 129 pgs: 129 active+clean; 158 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 982 KiB/s wr, 64 op/s 2024-03-18T20:29:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:42 smithi169 ceph-mon[48529]: pgmap v377: 129 pgs: 129 active+clean; 158 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 982 KiB/s wr, 64 op/s 2024-03-18T20:29:44.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:44 smithi136 ceph-mon[48073]: pgmap v378: 129 pgs: 129 active+clean; 158 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 778 KiB/s wr, 59 op/s 2024-03-18T20:29:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:29:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:44 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:44 smithi098 ceph-mon[43462]: pgmap v378: 129 pgs: 129 active+clean; 158 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 778 KiB/s wr, 59 op/s 2024-03-18T20:29:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:29:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:44 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:44 smithi169 ceph-mon[48529]: pgmap v378: 129 pgs: 129 active+clean; 158 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 778 KiB/s wr, 59 op/s 2024-03-18T20:29:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:29:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:44 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:29:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:29:45.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:29:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:29:46.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:29:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:29:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:29:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:46 smithi136 ceph-mon[48073]: pgmap v379: 129 pgs: 129 active+clean; 160 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 867 KiB/s wr, 60 op/s 2024-03-18T20:29:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:46 smithi098 ceph-mon[43462]: pgmap v379: 129 pgs: 129 active+clean; 160 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 867 KiB/s wr, 60 op/s 2024-03-18T20:29:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:46 smithi169 ceph-mon[48529]: pgmap v379: 129 pgs: 129 active+clean; 160 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 867 KiB/s wr, 60 op/s 2024-03-18T20:29:47.578 INFO:tasks.workunit.client.0.smithi098.stdout: 24 13138 85 8753 16 4651 128 2024-03-18T20:29:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:48 smithi136 ceph-mon[48073]: pgmap v380: 129 pgs: 129 active+clean; 162 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 833 KiB/s wr, 53 op/s 2024-03-18T20:29:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:48 smithi098 ceph-mon[43462]: pgmap v380: 129 pgs: 129 active+clean; 162 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 833 KiB/s wr, 53 op/s 2024-03-18T20:29:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:48 smithi169 ceph-mon[48529]: pgmap v380: 129 pgs: 129 active+clean; 162 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 833 KiB/s wr, 53 op/s 2024-03-18T20:29:50.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:50 smithi136 ceph-mon[48073]: pgmap v381: 129 pgs: 129 active+clean; 163 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 847 KiB/s wr, 52 op/s 2024-03-18T20:29:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:50 smithi098 ceph-mon[43462]: pgmap v381: 129 pgs: 129 active+clean; 163 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 847 KiB/s wr, 52 op/s 2024-03-18T20:29:51.141 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:50 smithi169 ceph-mon[48529]: pgmap v381: 129 pgs: 129 active+clean; 163 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 847 KiB/s wr, 52 op/s 2024-03-18T20:29:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:52 smithi098 ceph-mon[43462]: pgmap v382: 129 pgs: 129 active+clean; 164 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 945 KiB/s wr, 55 op/s 2024-03-18T20:29:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:52 smithi169 ceph-mon[48529]: pgmap v382: 129 pgs: 129 active+clean; 164 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 945 KiB/s wr, 55 op/s 2024-03-18T20:29:52.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:52 smithi136 ceph-mon[48073]: pgmap v382: 129 pgs: 129 active+clean; 164 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 945 KiB/s wr, 55 op/s 2024-03-18T20:29:54.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:54 smithi136 ceph-mon[48073]: pgmap v383: 129 pgs: 129 active+clean; 166 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 829 KiB/s wr, 51 op/s 2024-03-18T20:29:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:54 smithi098 ceph-mon[43462]: pgmap v383: 129 pgs: 129 active+clean; 166 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 829 KiB/s wr, 51 op/s 2024-03-18T20:29:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:54 smithi169 ceph-mon[48529]: pgmap v383: 129 pgs: 129 active+clean; 166 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 829 KiB/s wr, 51 op/s 2024-03-18T20:29:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:56 smithi136 ceph-mon[48073]: pgmap v384: 129 pgs: 129 active+clean; 167 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 820 KiB/s wr, 48 op/s 2024-03-18T20:29:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:56 smithi098 ceph-mon[43462]: pgmap v384: 129 pgs: 129 active+clean; 167 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 820 KiB/s wr, 48 op/s 2024-03-18T20:29:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:56 smithi169 ceph-mon[48529]: pgmap v384: 129 pgs: 129 active+clean; 167 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 820 KiB/s wr, 48 op/s 2024-03-18T20:29:57.578 INFO:tasks.workunit.client.0.smithi098.stdout: 25 13435 42 8766 45 3901 111 2024-03-18T20:29:58.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:29:58 smithi136 ceph-mon[48073]: pgmap v385: 129 pgs: 129 active+clean; 170 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 979 KiB/s wr, 50 op/s 2024-03-18T20:29:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:29:58 smithi098 ceph-mon[43462]: pgmap v385: 129 pgs: 129 active+clean; 170 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 979 KiB/s wr, 50 op/s 2024-03-18T20:29:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:29:58 smithi169 ceph-mon[48529]: pgmap v385: 129 pgs: 129 active+clean; 170 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 979 KiB/s wr, 50 op/s 2024-03-18T20:30:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:00 smithi136 ceph-mon[48073]: pgmap v386: 129 pgs: 129 active+clean; 171 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 792 KiB/s wr, 48 op/s 2024-03-18T20:30:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:00 smithi136 ceph-mon[48073]: overall HEALTH_OK 2024-03-18T20:30:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:00 smithi098 ceph-mon[43462]: pgmap v386: 129 pgs: 129 active+clean; 171 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 792 KiB/s wr, 48 op/s 2024-03-18T20:30:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:00 smithi098 ceph-mon[43462]: overall HEALTH_OK 2024-03-18T20:30:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:00 smithi169 ceph-mon[48529]: pgmap v386: 129 pgs: 129 active+clean; 171 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 792 KiB/s wr, 48 op/s 2024-03-18T20:30:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:00 smithi169 ceph-mon[48529]: overall HEALTH_OK 2024-03-18T20:30:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:02 smithi098 ceph-mon[43462]: pgmap v387: 129 pgs: 129 active+clean; 174 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-03-18T20:30:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:02 smithi169 ceph-mon[48529]: pgmap v387: 129 pgs: 129 active+clean; 174 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-03-18T20:30:02.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:02 smithi136 ceph-mon[48073]: pgmap v387: 129 pgs: 129 active+clean; 174 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 1.0 MiB/s wr, 55 op/s 2024-03-18T20:30:04.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:04 smithi136 ceph-mon[48073]: pgmap v388: 129 pgs: 129 active+clean; 175 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 967 KiB/s wr, 56 op/s 2024-03-18T20:30:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:04 smithi098 ceph-mon[43462]: pgmap v388: 129 pgs: 129 active+clean; 175 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 967 KiB/s wr, 56 op/s 2024-03-18T20:30:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:04 smithi169 ceph-mon[48529]: pgmap v388: 129 pgs: 129 active+clean; 175 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 967 KiB/s wr, 56 op/s 2024-03-18T20:30:06.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:06 smithi136 ceph-mon[48073]: pgmap v389: 129 pgs: 129 active+clean; 175 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 858 KiB/s wr, 53 op/s 2024-03-18T20:30:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:06 smithi098 ceph-mon[43462]: pgmap v389: 129 pgs: 129 active+clean; 175 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 858 KiB/s wr, 53 op/s 2024-03-18T20:30:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:06 smithi169 ceph-mon[48529]: pgmap v389: 129 pgs: 129 active+clean; 175 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 858 KiB/s wr, 53 op/s 2024-03-18T20:30:07.578 INFO:tasks.workunit.client.0.smithi098.stdout: 25 13295 36 8900 61 4389 132 2024-03-18T20:30:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:08 smithi136 ceph-mon[48073]: pgmap v390: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-03-18T20:30:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:08 smithi098 ceph-mon[43462]: pgmap v390: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-03-18T20:30:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:08 smithi169 ceph-mon[48529]: pgmap v390: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.0 MiB/s wr, 57 op/s 2024-03-18T20:30:10.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:10 smithi136 ceph-mon[48073]: pgmap v391: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 789 KiB/s wr, 48 op/s 2024-03-18T20:30:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:10 smithi098 ceph-mon[43462]: pgmap v391: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 789 KiB/s wr, 48 op/s 2024-03-18T20:30:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:10 smithi169 ceph-mon[48529]: pgmap v391: 129 pgs: 129 active+clean; 178 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 789 KiB/s wr, 48 op/s 2024-03-18T20:30:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:12 smithi098 ceph-mon[43462]: pgmap v392: 129 pgs: 129 active+clean; 182 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:30:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:12 smithi169 ceph-mon[48529]: pgmap v392: 129 pgs: 129 active+clean; 182 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:30:12.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:12 smithi136 ceph-mon[48073]: pgmap v392: 129 pgs: 129 active+clean; 182 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:30:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:14 smithi136 ceph-mon[48073]: pgmap v393: 129 pgs: 129 active+clean; 185 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-03-18T20:30:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:14 smithi098 ceph-mon[43462]: pgmap v393: 129 pgs: 129 active+clean; 185 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-03-18T20:30:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:14 smithi169 ceph-mon[48529]: pgmap v393: 129 pgs: 129 active+clean; 185 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-03-18T20:30:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:16 smithi136 ceph-mon[48073]: pgmap v394: 129 pgs: 129 active+clean; 186 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:16 smithi098 ceph-mon[43462]: pgmap v394: 129 pgs: 129 active+clean; 186 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:16 smithi169 ceph-mon[48529]: pgmap v394: 129 pgs: 129 active+clean; 186 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:17.579 INFO:tasks.workunit.client.0.smithi098.stdout: 26 12722 33 8981 103 4282 167 2024-03-18T20:30:18.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:18 smithi136 ceph-mon[48073]: pgmap v395: 129 pgs: 129 active+clean; 187 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-03-18T20:30:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:18 smithi098 ceph-mon[43462]: pgmap v395: 129 pgs: 129 active+clean; 187 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-03-18T20:30:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:18 smithi169 ceph-mon[48529]: pgmap v395: 129 pgs: 129 active+clean; 187 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-03-18T20:30:20.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:20 smithi136 ceph-mon[48073]: pgmap v396: 129 pgs: 129 active+clean; 189 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 60 op/s 2024-03-18T20:30:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:20 smithi098 ceph-mon[43462]: pgmap v396: 129 pgs: 129 active+clean; 189 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 60 op/s 2024-03-18T20:30:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:20 smithi169 ceph-mon[48529]: pgmap v396: 129 pgs: 129 active+clean; 189 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.1 MiB/s wr, 60 op/s 2024-03-18T20:30:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:22 smithi098 ceph-mon[43462]: pgmap v397: 129 pgs: 129 active+clean; 192 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-03-18T20:30:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:22 smithi169 ceph-mon[48529]: pgmap v397: 129 pgs: 129 active+clean; 192 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-03-18T20:30:22.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:22 smithi136 ceph-mon[48073]: pgmap v397: 129 pgs: 129 active+clean; 192 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-03-18T20:30:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:24 smithi136 ceph-mon[48073]: pgmap v398: 129 pgs: 129 active+clean; 193 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:30:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:24 smithi098 ceph-mon[43462]: pgmap v398: 129 pgs: 129 active+clean; 193 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:30:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:24 smithi169 ceph-mon[48529]: pgmap v398: 129 pgs: 129 active+clean; 193 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-18T20:30:26.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:26 smithi136 ceph-mon[48073]: pgmap v399: 129 pgs: 129 active+clean; 193 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-03-18T20:30:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:26 smithi098 ceph-mon[43462]: pgmap v399: 129 pgs: 129 active+clean; 193 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-03-18T20:30:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:26 smithi169 ceph-mon[48529]: pgmap v399: 129 pgs: 129 active+clean; 193 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 838 KiB/s wr, 63 op/s 2024-03-18T20:30:27.579 INFO:tasks.workunit.client.0.smithi098.stdout: 27 13198 93 8839 40 4024 159 2024-03-18T20:30:28.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:28 smithi136 ceph-mon[48073]: pgmap v400: 129 pgs: 129 active+clean; 197 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-03-18T20:30:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:28 smithi098 ceph-mon[43462]: pgmap v400: 129 pgs: 129 active+clean; 197 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-03-18T20:30:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:28 smithi169 ceph-mon[48529]: pgmap v400: 129 pgs: 129 active+clean; 197 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.0 MiB/s wr, 67 op/s 2024-03-18T20:30:30.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:30 smithi136 ceph-mon[48073]: pgmap v401: 129 pgs: 129 active+clean; 197 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 948 KiB/s wr, 59 op/s 2024-03-18T20:30:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:30 smithi098 ceph-mon[43462]: pgmap v401: 129 pgs: 129 active+clean; 197 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 948 KiB/s wr, 59 op/s 2024-03-18T20:30:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:30 smithi169 ceph-mon[48529]: pgmap v401: 129 pgs: 129 active+clean; 197 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 948 KiB/s wr, 59 op/s 2024-03-18T20:30:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:32 smithi098 ceph-mon[43462]: pgmap v402: 129 pgs: 129 active+clean; 201 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:32 smithi169 ceph-mon[48529]: pgmap v402: 129 pgs: 129 active+clean; 201 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:32 smithi136 ceph-mon[48073]: pgmap v402: 129 pgs: 129 active+clean; 201 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.1 MiB/s wr, 59 op/s 2024-03-18T20:30:34.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:34 smithi136 ceph-mon[48073]: pgmap v403: 129 pgs: 129 active+clean; 202 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 910 KiB/s wr, 51 op/s 2024-03-18T20:30:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:34 smithi098 ceph-mon[43462]: pgmap v403: 129 pgs: 129 active+clean; 202 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 910 KiB/s wr, 51 op/s 2024-03-18T20:30:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:34 smithi169 ceph-mon[48529]: pgmap v403: 129 pgs: 129 active+clean; 202 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 910 KiB/s wr, 51 op/s 2024-03-18T20:30:36.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:36 smithi136 ceph-mon[48073]: pgmap v404: 129 pgs: 129 active+clean; 202 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 889 KiB/s wr, 46 op/s 2024-03-18T20:30:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:36 smithi098 ceph-mon[43462]: pgmap v404: 129 pgs: 129 active+clean; 202 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 889 KiB/s wr, 46 op/s 2024-03-18T20:30:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:36 smithi169 ceph-mon[48529]: pgmap v404: 129 pgs: 129 active+clean; 202 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 889 KiB/s wr, 46 op/s 2024-03-18T20:30:37.579 INFO:tasks.workunit.client.0.smithi098.stdout: 28 12175 95 9172 17 5077 119 2024-03-18T20:30:38.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:38 smithi136 ceph-mon[48073]: pgmap v405: 129 pgs: 129 active+clean; 205 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:30:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:38 smithi098 ceph-mon[43462]: pgmap v405: 129 pgs: 129 active+clean; 205 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:30:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:38 smithi169 ceph-mon[48529]: pgmap v405: 129 pgs: 129 active+clean; 205 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:30:40.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:40 smithi136 ceph-mon[48073]: pgmap v406: 129 pgs: 129 active+clean; 205 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 755 KiB/s wr, 36 op/s 2024-03-18T20:30:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:40 smithi098 ceph-mon[43462]: pgmap v406: 129 pgs: 129 active+clean; 205 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 755 KiB/s wr, 36 op/s 2024-03-18T20:30:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:40 smithi169 ceph-mon[48529]: pgmap v406: 129 pgs: 129 active+clean; 205 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 755 KiB/s wr, 36 op/s 2024-03-18T20:30:42.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:42 smithi098 ceph-mon[43462]: pgmap v407: 129 pgs: 129 active+clean; 207 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 957 KiB/s wr, 43 op/s 2024-03-18T20:30:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:42 smithi169 ceph-mon[48529]: pgmap v407: 129 pgs: 129 active+clean; 207 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 957 KiB/s wr, 43 op/s 2024-03-18T20:30:42.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:42 smithi136 ceph-mon[48073]: pgmap v407: 129 pgs: 129 active+clean; 207 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 957 KiB/s wr, 43 op/s 2024-03-18T20:30:44.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:44 smithi136 ceph-mon[48073]: pgmap v408: 129 pgs: 129 active+clean; 209 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 741 KiB/s wr, 44 op/s 2024-03-18T20:30:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:44 smithi098 ceph-mon[43462]: pgmap v408: 129 pgs: 129 active+clean; 209 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 741 KiB/s wr, 44 op/s 2024-03-18T20:30:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:44 smithi169 ceph-mon[48529]: pgmap v408: 129 pgs: 129 active+clean; 209 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 741 KiB/s wr, 44 op/s 2024-03-18T20:30:45.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:45 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:30:46.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:45 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:30:46.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:45 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:30:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: pgmap v409: 129 pgs: 129 active+clean; 209 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 738 KiB/s wr, 45 op/s 2024-03-18T20:30:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:30:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:30:46.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: pgmap v409: 129 pgs: 129 active+clean; 209 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 738 KiB/s wr, 45 op/s 2024-03-18T20:30:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:30:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:30:47.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: pgmap v409: 129 pgs: 129 active+clean; 209 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 738 KiB/s wr, 45 op/s 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:30:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:30:47.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:30:47.579 INFO:tasks.workunit.client.0.smithi098.stdout: 28 12138 89 8822 26 5053 125 2024-03-18T20:30:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:48 smithi136 ceph-mon[48073]: pgmap v410: 129 pgs: 129 active+clean; 213 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1006 KiB/s wr, 52 op/s 2024-03-18T20:30:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:48 smithi098 ceph-mon[43462]: pgmap v410: 129 pgs: 129 active+clean; 213 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1006 KiB/s wr, 52 op/s 2024-03-18T20:30:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:48 smithi169 ceph-mon[48529]: pgmap v410: 129 pgs: 129 active+clean; 213 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1006 KiB/s wr, 52 op/s 2024-03-18T20:30:50.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:50 smithi136 ceph-mon[48073]: pgmap v411: 129 pgs: 129 active+clean; 213 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 798 KiB/s wr, 45 op/s 2024-03-18T20:30:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:50 smithi098 ceph-mon[43462]: pgmap v411: 129 pgs: 129 active+clean; 213 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 798 KiB/s wr, 45 op/s 2024-03-18T20:30:51.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:50 smithi169 ceph-mon[48529]: pgmap v411: 129 pgs: 129 active+clean; 213 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 798 KiB/s wr, 45 op/s 2024-03-18T20:30:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:52 smithi098 ceph-mon[43462]: pgmap v412: 129 pgs: 129 active+clean; 217 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-03-18T20:30:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:52 smithi169 ceph-mon[48529]: pgmap v412: 129 pgs: 129 active+clean; 217 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-03-18T20:30:52.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:52 smithi136 ceph-mon[48073]: pgmap v412: 129 pgs: 129 active+clean; 217 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.1 MiB/s wr, 57 op/s 2024-03-18T20:30:54.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:54 smithi136 ceph-mon[48073]: pgmap v413: 129 pgs: 129 active+clean; 218 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-03-18T20:30:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:54 smithi098 ceph-mon[43462]: pgmap v413: 129 pgs: 129 active+clean; 218 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-03-18T20:30:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:54 smithi169 ceph-mon[48529]: pgmap v413: 129 pgs: 129 active+clean; 218 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.0 MiB/s wr, 58 op/s 2024-03-18T20:30:56.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:56 smithi136 ceph-mon[48073]: pgmap v414: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.2 MiB/s wr, 66 op/s 2024-03-18T20:30:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:56 smithi098 ceph-mon[43462]: pgmap v414: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.2 MiB/s wr, 66 op/s 2024-03-18T20:30:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:56 smithi169 ceph-mon[48529]: pgmap v414: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 1.2 MiB/s wr, 66 op/s 2024-03-18T20:30:57.579 INFO:tasks.workunit.client.0.smithi098.stdout: 29 12392 75 7950 49 5296 158 2024-03-18T20:30:58.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:30:58 smithi136 ceph-mon[48073]: pgmap v415: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.2 MiB/s wr, 72 op/s 2024-03-18T20:30:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:30:58 smithi098 ceph-mon[43462]: pgmap v415: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.2 MiB/s wr, 72 op/s 2024-03-18T20:30:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:30:58 smithi169 ceph-mon[48529]: pgmap v415: 129 pgs: 129 active+clean; 222 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 1.2 MiB/s wr, 72 op/s 2024-03-18T20:31:00.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:00 smithi136 ceph-mon[48073]: pgmap v416: 129 pgs: 129 active+clean; 224 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:31:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:00 smithi098 ceph-mon[43462]: pgmap v416: 129 pgs: 129 active+clean; 224 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:31:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:00 smithi169 ceph-mon[48529]: pgmap v416: 129 pgs: 129 active+clean; 224 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.1 MiB/s wr, 67 op/s 2024-03-18T20:31:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:02 smithi098 ceph-mon[43462]: pgmap v417: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-03-18T20:31:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:02 smithi169 ceph-mon[48529]: pgmap v417: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-03-18T20:31:02.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:02 smithi136 ceph-mon[48073]: pgmap v417: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-03-18T20:31:04.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:04 smithi136 ceph-mon[48073]: pgmap v418: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 MiB/s wr, 72 op/s 2024-03-18T20:31:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:04 smithi098 ceph-mon[43462]: pgmap v418: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 MiB/s wr, 72 op/s 2024-03-18T20:31:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:04 smithi169 ceph-mon[48529]: pgmap v418: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 MiB/s wr, 72 op/s 2024-03-18T20:31:06.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:06 smithi136 ceph-mon[48073]: pgmap v419: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-03-18T20:31:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:06 smithi098 ceph-mon[43462]: pgmap v419: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-03-18T20:31:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:06 smithi169 ceph-mon[48529]: pgmap v419: 129 pgs: 129 active+clean; 228 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-03-18T20:31:07.579 INFO:tasks.workunit.client.0.smithi098.stdout: 29 12308 65 7889 29 6134 127 2024-03-18T20:31:08.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:08 smithi136 ceph-mon[48073]: pgmap v420: 129 pgs: 129 active+clean; 232 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:31:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:08 smithi098 ceph-mon[43462]: pgmap v420: 129 pgs: 129 active+clean; 232 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:31:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:08 smithi169 ceph-mon[48529]: pgmap v420: 129 pgs: 129 active+clean; 232 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 1.0 MiB/s wr, 61 op/s 2024-03-18T20:31:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:10 smithi098 ceph-mon[43462]: pgmap v421: 129 pgs: 129 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1010 KiB/s wr, 52 op/s 2024-03-18T20:31:11.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:10 smithi169 ceph-mon[48529]: pgmap v421: 129 pgs: 129 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1010 KiB/s wr, 52 op/s 2024-03-18T20:31:11.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:10 smithi136 ceph-mon[48073]: pgmap v421: 129 pgs: 129 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1010 KiB/s wr, 52 op/s 2024-03-18T20:31:12.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:12 smithi098 ceph-mon[43462]: pgmap v422: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-03-18T20:31:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:12 smithi169 ceph-mon[48529]: pgmap v422: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-03-18T20:31:12.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:12 smithi136 ceph-mon[48073]: pgmap v422: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-03-18T20:31:14.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:14 smithi136 ceph-mon[48073]: pgmap v423: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 958 KiB/s wr, 50 op/s 2024-03-18T20:31:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:14 smithi098 ceph-mon[43462]: pgmap v423: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 958 KiB/s wr, 50 op/s 2024-03-18T20:31:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:14 smithi169 ceph-mon[48529]: pgmap v423: 129 pgs: 129 active+clean; 237 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 958 KiB/s wr, 50 op/s 2024-03-18T20:31:16.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:16 smithi136 ceph-mon[48073]: pgmap v424: 129 pgs: 129 active+clean; 239 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:31:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:16 smithi098 ceph-mon[43462]: pgmap v424: 129 pgs: 129 active+clean; 239 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:31:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:16 smithi169 ceph-mon[48529]: pgmap v424: 129 pgs: 129 active+clean; 239 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.0 MiB/s wr, 47 op/s 2024-03-18T20:31:17.579 INFO:tasks.workunit.client.0.smithi098.stdout: 29 12647 43 8838 38 4852 110 2024-03-18T20:31:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:18 smithi098 ceph-mon[43462]: pgmap v425: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:18 smithi169 ceph-mon[48529]: pgmap v425: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:19.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:18 smithi136 ceph-mon[48073]: pgmap v425: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:20 smithi098 ceph-mon[43462]: pgmap v426: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 837 KiB/s wr, 43 op/s 2024-03-18T20:31:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:20 smithi169 ceph-mon[48529]: pgmap v426: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 837 KiB/s wr, 43 op/s 2024-03-18T20:31:21.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:20 smithi136 ceph-mon[48073]: pgmap v426: 129 pgs: 129 active+clean; 241 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 837 KiB/s wr, 43 op/s 2024-03-18T20:31:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:22 smithi098 ceph-mon[43462]: pgmap v427: 129 pgs: 129 active+clean; 246 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.2 MiB/s wr, 50 op/s 2024-03-18T20:31:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:22 smithi169 ceph-mon[48529]: pgmap v427: 129 pgs: 129 active+clean; 246 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.2 MiB/s wr, 50 op/s 2024-03-18T20:31:22.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:22 smithi136 ceph-mon[48073]: pgmap v427: 129 pgs: 129 active+clean; 246 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.2 MiB/s wr, 50 op/s 2024-03-18T20:31:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:24 smithi136 ceph-mon[48073]: pgmap v428: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 922 KiB/s wr, 45 op/s 2024-03-18T20:31:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:24 smithi098 ceph-mon[43462]: pgmap v428: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 922 KiB/s wr, 45 op/s 2024-03-18T20:31:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:24 smithi169 ceph-mon[48529]: pgmap v428: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 922 KiB/s wr, 45 op/s 2024-03-18T20:31:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:26 smithi098 ceph-mon[43462]: pgmap v429: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 914 KiB/s wr, 44 op/s 2024-03-18T20:31:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:26 smithi169 ceph-mon[48529]: pgmap v429: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 914 KiB/s wr, 44 op/s 2024-03-18T20:31:27.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:26 smithi136 ceph-mon[48073]: pgmap v429: 129 pgs: 129 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 914 KiB/s wr, 44 op/s 2024-03-18T20:31:27.579 INFO:tasks.workunit.client.0.smithi098.stdout: 32 12981 54 8698 32 4442 82 2024-03-18T20:31:29.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:28 smithi098 ceph-mon[43462]: pgmap v430: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1013 KiB/s wr, 46 op/s 2024-03-18T20:31:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:28 smithi169 ceph-mon[48529]: pgmap v430: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1013 KiB/s wr, 46 op/s 2024-03-18T20:31:29.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:28 smithi136 ceph-mon[48073]: pgmap v430: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1013 KiB/s wr, 46 op/s 2024-03-18T20:31:31.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:30 smithi098 ceph-mon[43462]: pgmap v431: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 851 KiB/s wr, 38 op/s 2024-03-18T20:31:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:30 smithi169 ceph-mon[48529]: pgmap v431: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 851 KiB/s wr, 38 op/s 2024-03-18T20:31:31.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:30 smithi136 ceph-mon[48073]: pgmap v431: 129 pgs: 129 active+clean; 250 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 851 KiB/s wr, 38 op/s 2024-03-18T20:31:32.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:32 smithi098 ceph-mon[43462]: pgmap v432: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:32 smithi169 ceph-mon[48529]: pgmap v432: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:32 smithi136 ceph-mon[48073]: pgmap v432: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-18T20:31:35.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:34 smithi098 ceph-mon[43462]: pgmap v433: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 912 KiB/s wr, 47 op/s 2024-03-18T20:31:35.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:34 smithi169 ceph-mon[48529]: pgmap v433: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 912 KiB/s wr, 47 op/s 2024-03-18T20:31:35.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:34 smithi136 ceph-mon[48073]: pgmap v433: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 912 KiB/s wr, 47 op/s 2024-03-18T20:31:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:36 smithi098 ceph-mon[43462]: pgmap v434: 129 pgs: 129 active+clean; 256 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 918 KiB/s wr, 44 op/s 2024-03-18T20:31:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:36 smithi169 ceph-mon[48529]: pgmap v434: 129 pgs: 129 active+clean; 256 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 918 KiB/s wr, 44 op/s 2024-03-18T20:31:37.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:36 smithi136 ceph-mon[48073]: pgmap v434: 129 pgs: 129 active+clean; 256 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 918 KiB/s wr, 44 op/s 2024-03-18T20:31:37.579 INFO:tasks.workunit.client.0.smithi098.stdout: 32 13120 66 8738 3 4603 88 2024-03-18T20:31:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:38 smithi098 ceph-mon[43462]: pgmap v435: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 47 op/s 2024-03-18T20:31:39.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:38 smithi169 ceph-mon[48529]: pgmap v435: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 47 op/s 2024-03-18T20:31:39.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:38 smithi136 ceph-mon[48073]: pgmap v435: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 47 op/s 2024-03-18T20:31:41.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:40 smithi098 ceph-mon[43462]: pgmap v436: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 871 KiB/s wr, 40 op/s 2024-03-18T20:31:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:40 smithi169 ceph-mon[48529]: pgmap v436: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 871 KiB/s wr, 40 op/s 2024-03-18T20:31:41.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:40 smithi136 ceph-mon[48073]: pgmap v436: 129 pgs: 129 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 871 KiB/s wr, 40 op/s 2024-03-18T20:31:42.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:42 smithi098 ceph-mon[43462]: pgmap v437: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-18T20:31:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:42 smithi169 ceph-mon[48529]: pgmap v437: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-18T20:31:42.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:42 smithi136 ceph-mon[48073]: pgmap v437: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-18T20:31:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:44 smithi098 ceph-mon[43462]: pgmap v438: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 914 KiB/s wr, 38 op/s 2024-03-18T20:31:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:44 smithi169 ceph-mon[48529]: pgmap v438: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 914 KiB/s wr, 38 op/s 2024-03-18T20:31:45.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:44 smithi136 ceph-mon[48073]: pgmap v438: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 914 KiB/s wr, 38 op/s 2024-03-18T20:31:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:46 smithi098 ceph-mon[43462]: pgmap v439: 129 pgs: 129 active+clean; 268 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 1.1 MiB/s wr, 38 op/s 2024-03-18T20:31:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:46 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:31:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:46 smithi169 ceph-mon[48529]: pgmap v439: 129 pgs: 129 active+clean; 268 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 1.1 MiB/s wr, 38 op/s 2024-03-18T20:31:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:46 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:31:47.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:46 smithi136 ceph-mon[48073]: pgmap v439: 129 pgs: 129 active+clean; 268 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 1.1 MiB/s wr, 38 op/s 2024-03-18T20:31:47.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:46 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:31:47.579 INFO:tasks.workunit.client.0.smithi098.stdout: 32 12315 66 8239 9 5223 91 2024-03-18T20:31:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:31:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:31:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:31:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:31:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:31:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:31:48.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:31:48.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:31:48.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:31:49.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:48 smithi136 ceph-mon[48073]: pgmap v440: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.2 MiB/s wr, 42 op/s 2024-03-18T20:31:49.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:48 smithi098 ceph-mon[43462]: pgmap v440: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.2 MiB/s wr, 42 op/s 2024-03-18T20:31:49.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:48 smithi169 ceph-mon[48529]: pgmap v440: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 1.2 MiB/s wr, 42 op/s 2024-03-18T20:31:51.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:50 smithi136 ceph-mon[48073]: pgmap v441: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-03-18T20:31:51.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:50 smithi098 ceph-mon[43462]: pgmap v441: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-03-18T20:31:51.383 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:50 smithi169 ceph-mon[48529]: pgmap v441: 129 pgs: 129 active+clean; 270 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-03-18T20:31:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:52 smithi098 ceph-mon[43462]: pgmap v442: 129 pgs: 129 active+clean; 271 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-03-18T20:31:52.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:52 smithi169 ceph-mon[48529]: pgmap v442: 129 pgs: 129 active+clean; 271 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-03-18T20:31:52.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:52 smithi136 ceph-mon[48073]: pgmap v442: 129 pgs: 129 active+clean; 271 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-03-18T20:31:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:54 smithi098 ceph-mon[43462]: pgmap v443: 129 pgs: 129 active+clean; 273 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 941 KiB/s wr, 44 op/s 2024-03-18T20:31:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:54 smithi169 ceph-mon[48529]: pgmap v443: 129 pgs: 129 active+clean; 273 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 941 KiB/s wr, 44 op/s 2024-03-18T20:31:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:54 smithi136 ceph-mon[48073]: pgmap v443: 129 pgs: 129 active+clean; 273 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 941 KiB/s wr, 44 op/s 2024-03-18T20:31:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:56 smithi098 ceph-mon[43462]: pgmap v444: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-03-18T20:31:57.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:56 smithi169 ceph-mon[48529]: pgmap v444: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-03-18T20:31:57.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:56 smithi136 ceph-mon[48073]: pgmap v444: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-03-18T20:31:57.579 INFO:tasks.workunit.client.0.smithi098.stdout: 32 12445 38 7571 37 5588 92 2024-03-18T20:31:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:31:58 smithi098 ceph-mon[43462]: pgmap v445: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 939 KiB/s wr, 45 op/s 2024-03-18T20:31:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:31:58 smithi169 ceph-mon[48529]: pgmap v445: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 939 KiB/s wr, 45 op/s 2024-03-18T20:31:59.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:31:58 smithi136 ceph-mon[48073]: pgmap v445: 129 pgs: 129 active+clean; 278 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 939 KiB/s wr, 45 op/s 2024-03-18T20:32:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:00 smithi098 ceph-mon[43462]: pgmap v446: 129 pgs: 129 active+clean; 282 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.1 MiB/s wr, 42 op/s 2024-03-18T20:32:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:00 smithi169 ceph-mon[48529]: pgmap v446: 129 pgs: 129 active+clean; 282 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.1 MiB/s wr, 42 op/s 2024-03-18T20:32:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:00 smithi136 ceph-mon[48073]: pgmap v446: 129 pgs: 129 active+clean; 282 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.1 MiB/s wr, 42 op/s 2024-03-18T20:32:02.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:02 smithi098 ceph-mon[43462]: pgmap v447: 129 pgs: 129 active+clean; 283 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-18T20:32:02.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:02 smithi169 ceph-mon[48529]: pgmap v447: 129 pgs: 129 active+clean; 283 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-18T20:32:02.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:02 smithi136 ceph-mon[48073]: pgmap v447: 129 pgs: 129 active+clean; 283 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-18T20:32:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:04 smithi098 ceph-mon[43462]: pgmap v448: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-03-18T20:32:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:04 smithi169 ceph-mon[48529]: pgmap v448: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-03-18T20:32:05.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:04 smithi136 ceph-mon[48073]: pgmap v448: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-03-18T20:32:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:06 smithi098 ceph-mon[43462]: pgmap v449: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-03-18T20:32:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:06 smithi169 ceph-mon[48529]: pgmap v449: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-03-18T20:32:07.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:06 smithi136 ceph-mon[48073]: pgmap v449: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 64 op/s 2024-03-18T20:32:07.579 INFO:tasks.workunit.client.0.smithi098.stdout: 32 12112 22 7885 52 4956 82 2024-03-18T20:32:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:08 smithi098 ceph-mon[43462]: pgmap v450: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 973 KiB/s wr, 118 op/s 2024-03-18T20:32:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:08 smithi169 ceph-mon[48529]: pgmap v450: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 973 KiB/s wr, 118 op/s 2024-03-18T20:32:09.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:08 smithi136 ceph-mon[48073]: pgmap v450: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 973 KiB/s wr, 118 op/s 2024-03-18T20:32:11.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:10 smithi098 ceph-mon[43462]: pgmap v451: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-03-18T20:32:11.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:10 smithi169 ceph-mon[48529]: pgmap v451: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-03-18T20:32:11.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:10 smithi136 ceph-mon[48073]: pgmap v451: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-03-18T20:32:11.458 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-03-18T20:32:11.458 INFO:tasks.workunit.client.0.smithi098.stdout:Final score for writes: 32 2024-03-18T20:32:11.458 INFO:tasks.workunit.client.0.smithi098.stdout:Final score for reads : 6571 2024-03-18T20:32:11.458 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-03-18T20:32:11.459 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T20:32:11.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T20:32:12.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:12 smithi098 ceph-mon[43462]: pgmap v452: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.3 MiB/s wr, 255 op/s 2024-03-18T20:32:12.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:12 smithi169 ceph-mon[48529]: pgmap v452: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.3 MiB/s wr, 255 op/s 2024-03-18T20:32:12.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:12 smithi136 ceph-mon[48073]: pgmap v452: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.3 MiB/s wr, 255 op/s 2024-03-18T20:32:15.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:14 smithi098 ceph-mon[43462]: pgmap v453: 129 pgs: 129 active+clean; 293 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-18T20:32:15.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:14 smithi169 ceph-mon[48529]: pgmap v453: 129 pgs: 129 active+clean; 293 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-18T20:32:15.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:14 smithi136 ceph-mon[48073]: pgmap v453: 129 pgs: 129 active+clean; 293 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-18T20:32:17.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:16 smithi098 ceph-mon[43462]: pgmap v454: 129 pgs: 129 active+clean; 287 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.8 MiB/s wr, 445 op/s 2024-03-18T20:32:17.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:16 smithi169 ceph-mon[48529]: pgmap v454: 129 pgs: 129 active+clean; 287 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.8 MiB/s wr, 445 op/s 2024-03-18T20:32:17.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:16 smithi136 ceph-mon[48073]: pgmap v454: 129 pgs: 129 active+clean; 287 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.8 MiB/s wr, 445 op/s 2024-03-18T20:32:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:18 smithi098 ceph-mon[43462]: pgmap v455: 129 pgs: 129 active+clean; 276 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.6 MiB/s wr, 522 op/s 2024-03-18T20:32:19.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:18 smithi169 ceph-mon[48529]: pgmap v455: 129 pgs: 129 active+clean; 276 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.6 MiB/s wr, 522 op/s 2024-03-18T20:32:19.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:18 smithi136 ceph-mon[48073]: pgmap v455: 129 pgs: 129 active+clean; 276 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.6 MiB/s wr, 522 op/s 2024-03-18T20:32:21.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:20 smithi098 ceph-mon[43462]: pgmap v456: 129 pgs: 129 active+clean; 271 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 528 op/s 2024-03-18T20:32:21.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:20 smithi169 ceph-mon[48529]: pgmap v456: 129 pgs: 129 active+clean; 271 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 528 op/s 2024-03-18T20:32:21.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:20 smithi136 ceph-mon[48073]: pgmap v456: 129 pgs: 129 active+clean; 271 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 528 op/s 2024-03-18T20:32:22.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:22 smithi098 ceph-mon[43462]: pgmap v457: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 586 op/s 2024-03-18T20:32:22.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:22 smithi169 ceph-mon[48529]: pgmap v457: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 586 op/s 2024-03-18T20:32:22.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:22 smithi136 ceph-mon[48073]: pgmap v457: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 586 op/s 2024-03-18T20:32:25.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:24 smithi098 ceph-mon[43462]: pgmap v458: 129 pgs: 129 active+clean; 246 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 584 op/s 2024-03-18T20:32:25.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:24 smithi169 ceph-mon[48529]: pgmap v458: 129 pgs: 129 active+clean; 246 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 584 op/s 2024-03-18T20:32:25.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:24 smithi136 ceph-mon[48073]: pgmap v458: 129 pgs: 129 active+clean; 246 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 584 op/s 2024-03-18T20:32:27.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:26 smithi098 ceph-mon[43462]: pgmap v459: 129 pgs: 129 active+clean; 233 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-03-18T20:32:27.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:26 smithi169 ceph-mon[48529]: pgmap v459: 129 pgs: 129 active+clean; 233 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-03-18T20:32:27.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:26 smithi136 ceph-mon[48073]: pgmap v459: 129 pgs: 129 active+clean; 233 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-03-18T20:32:29.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:28 smithi098 ceph-mon[43462]: pgmap v460: 129 pgs: 129 active+clean; 220 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 551 op/s 2024-03-18T20:32:29.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:28 smithi169 ceph-mon[48529]: pgmap v460: 129 pgs: 129 active+clean; 220 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 551 op/s 2024-03-18T20:32:29.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:28 smithi136 ceph-mon[48073]: pgmap v460: 129 pgs: 129 active+clean; 220 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 551 op/s 2024-03-18T20:32:31.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:30 smithi098 ceph-mon[43462]: pgmap v461: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 516 op/s 2024-03-18T20:32:31.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:30 smithi169 ceph-mon[48529]: pgmap v461: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 516 op/s 2024-03-18T20:32:31.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:30 smithi136 ceph-mon[48073]: pgmap v461: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 516 op/s 2024-03-18T20:32:31.547 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-18T20:32:31.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T20:32:31.825 DEBUG:teuthology.parallel:result is None 2024-03-18T20:32:31.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T20:32:31.867 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T20:32:31.867 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T20:32:31.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:32:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- 172.21.15.98:0/3977215979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 msgr2=0x7f8b681022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/3977215979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b681022e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8b580099c0 tx=0x7f8b58009cd0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- 172.21.15.98:0/3977215979 shutdown_connections 2024-03-18T20:32:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/3977215979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68102820 0x7f8b681090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/3977215979 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68100c90 0x7f8b681010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/3977215979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b681022e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- 172.21.15.98:0/3977215979 >> 172.21.15.98:0/3977215979 conn(0x7f8b680fc200 msgr2=0x7f8b680fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- 172.21.15.98:0/3977215979 shutdown_connections 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- 172.21.15.98:0/3977215979 wait complete. 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 Processor -- start 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.035+0000 7f8b6e96f700 1 -- start start 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68100c90 0x7f8b68196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68102820 0x7f8b6819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6819e620 con 0x7f8b68101e80 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6819e760 con 0x7f8b68100c90 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6819e8f0 con 0x7f8b68102820 2024-03-18T20:32:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b67fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68100c90 0x7f8b68196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52178/0 (socket says 172.21.15.98:52178) 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 -- 172.21.15.98:0/4117691820 learned_addr learned my addr 172.21.15.98:0/4117691820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6cf0c700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68102820 0x7f8b6819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 -- 172.21.15.98:0/4117691820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68102820 msgr2=0x7f8b6819e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68102820 0x7f8b6819e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 -- 172.21.15.98:0/4117691820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68100c90 msgr2=0x7f8b68196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68100c90 0x7f8b68196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 -- 172.21.15.98:0/4117691820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580095f0 con 0x7f8b68101e80 2024-03-18T20:32:32.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b677fe700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8b58000c00 tx=0x7f8b5800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5801d070 con 0x7f8b68101e80 2024-03-18T20:32:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6819eb70 con 0x7f8b68101e80 2024-03-18T20:32:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b5800bd30 con 0x7f8b68101e80 2024-03-18T20:32:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b580173f0 con 0x7f8b68101e80 2024-03-18T20:32:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.036+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6819ef90 con 0x7f8b68101e80 2024-03-18T20:32:32.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.037+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68108160 con 0x7f8b68101e80 2024-03-18T20:32:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.038+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8b58022470 con 0x7f8b68101e80 2024-03-18T20:32:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.039+0000 7f8b657fa700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 0x7f8b500a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.039+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f8b580cdeb0 con 0x7f8b68101e80 2024-03-18T20:32:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.039+0000 7f8b67fff700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 0x7f8b500a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.039+0000 7f8b67fff700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 0x7f8b500a35e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b68101ce0 tx=0x7f8b5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.043+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b58097310 con 0x7f8b68101e80 2024-03-18T20:32:32.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.341+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b68067e60 con 0x7f8b68101e80 2024-03-18T20:32:32.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.342+0000 7f8b657fa700 1 -- 172.21.15.98:0/4117691820 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24770 (secure 0 0 0) 0x7f8b5809a930 con 0x7f8b68101e80 2024-03-18T20:32:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:32:32.346 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:26:46.747051+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:32:32.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 msgr2=0x7f8b500a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 0x7f8b500a35e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b68101ce0 tx=0x7f8b5c009260 comp rx=0 tx=0).stop 2024-03-18T20:32:32.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 msgr2=0x7f8b68197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8b58000c00 tx=0x7f8b5800fa00 comp rx=0 tx=0).stop 2024-03-18T20:32:32.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 shutdown_connections 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b68102820 0x7f8b6819e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f8b500a1120 0x7f8b500a35e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b68100c90 0x7f8b68196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 --2- 172.21.15.98:0/4117691820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b68101e80 0x7f8b68197030 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 >> 172.21.15.98:0/4117691820 conn(0x7f8b680fc200 msgr2=0x7f8b681052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 shutdown_connections 2024-03-18T20:32:32.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.345+0000 7f8b6e96f700 1 -- 172.21.15.98:0/4117691820 wait complete. 2024-03-18T20:32:32.361 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:32:32.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:32:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 -- 172.21.15.98:0/1208796117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a84104110 msgr2=0x7f1a84106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/1208796117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a84104110 0x7f1a84106500 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009990 tx=0x7f1a6c009ca0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 -- 172.21.15.98:0/1208796117 shutdown_connections 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/1208796117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84106a40 0x7f1a84108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/1208796117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a841017e0 0x7f1a84103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/1208796117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a84104110 0x7f1a84106500 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 -- 172.21.15.98:0/1208796117 >> 172.21.15.98:0/1208796117 conn(0x7f1a840fb1d0 msgr2=0x7f1a840fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 -- 172.21.15.98:0/1208796117 shutdown_connections 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 -- 172.21.15.98:0/1208796117 wait complete. 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.484+0000 7f1a8a02a700 1 Processor -- start 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 -- start start 2024-03-18T20:32:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84104110 0x7f1a840ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a84106a40 0x7f1a840ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84100470 con 0x7f1a841017e0 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a841005e0 con 0x7f1a84104110 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a8a02a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a841a0b80 con 0x7f1a84106a40 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52208/0 (socket says 172.21.15.98:52208) 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 -- 172.21.15.98:0/401738604 learned_addr learned my addr 172.21.15.98:0/401738604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a82ffd700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84104110 0x7f1a840ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a83fff700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a84106a40 0x7f1a840ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 -- 172.21.15.98:0/401738604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a84106a40 msgr2=0x7f1a840ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a84106a40 0x7f1a840ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 -- 172.21.15.98:0/401738604 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84104110 msgr2=0x7f1a840ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84104110 0x7f1a840ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 -- 172.21.15.98:0/401738604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0095f0 con 0x7f1a841017e0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a837fe700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c9b0 tx=0x7f1a7400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.485+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74007aa0 con 0x7f1a841017e0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.486+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a841a0e10 con 0x7f1a841017e0 2024-03-18T20:32:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.486+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1a74007c00 con 0x7f1a841017e0 2024-03-18T20:32:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.486+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74018420 con 0x7f1a841017e0 2024-03-18T20:32:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.486+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a841a1240 con 0x7f1a841017e0 2024-03-18T20:32:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.486+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a840fcd50 con 0x7f1a841017e0 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.488+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f1a7400f450 con 0x7f1a841017e0 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.488+0000 7f1a80ff9700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 0x7f1a700a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.488+0000 7f1a82ffd700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 0x7f1a700a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.488+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f1a740cce60 con 0x7f1a841017e0 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.488+0000 7f1a82ffd700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 0x7f1a700a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.491+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1a740962c0 con 0x7f1a841017e0 2024-03-18T20:32:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.689+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a8402cee0 con 0x7f1a841017e0 2024-03-18T20:32:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.689+0000 7f1a80ff9700 1 -- 172.21.15.98:0/401738604 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a7400f700 con 0x7f1a841017e0 2024-03-18T20:32:32.693 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 msgr2=0x7f1a700a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 0x7f1a700a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c005fb0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 msgr2=0x7f1a84101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c9b0 tx=0x7f1a7400ccc0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 shutdown_connections 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a84106a40 0x7f1a840ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a841017e0 0x7f1a84101100 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f1a700a1120 0x7f1a700a35e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 --2- 172.21.15.98:0/401738604 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a84104110 0x7f1a840ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 >> 172.21.15.98:0/401738604 conn(0x7f1a840fb1d0 msgr2=0x7f1a84103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 shutdown_connections 2024-03-18T20:32:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.691+0000 7f1a8a02a700 1 -- 172.21.15.98:0/401738604 wait complete. 2024-03-18T20:32:32.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:32 smithi098 ceph-mon[43462]: pgmap v462: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 552 op/s 2024-03-18T20:32:32.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:32 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4117691820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:32:32.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:32:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 -- 172.21.15.98:0/2936786081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310073790 msgr2=0x7f4310073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 --2- 172.21.15.98:0/2936786081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310073790 0x7f4310073c10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4300009990 tx=0x7f4300009ca0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 -- 172.21.15.98:0/2936786081 shutdown_connections 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 --2- 172.21.15.98:0/2936786081 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310108c70 0x7f431010b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 --2- 172.21.15.98:0/2936786081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310074d70 0x7f43100731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 --2- 172.21.15.98:0/2936786081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310073790 0x7f4310073c10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 -- 172.21.15.98:0/2936786081 >> 172.21.15.98:0/2936786081 conn(0x7f43100fc2f0 msgr2=0x7f43100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 -- 172.21.15.98:0/2936786081 shutdown_connections 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 -- 172.21.15.98:0/2936786081 wait complete. 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.829+0000 7f4314d5d700 1 Processor -- start 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 -- start start 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310073790 0x7f4310198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310108c70 0x7f43101a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43101a08e0 con 0x7f4310074d70 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43101a0a50 con 0x7f4310108c70 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43101a0c10 con 0x7f4310073790 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52216/0 (socket says 172.21.15.98:52216) 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 -- 172.21.15.98:0/1374895065 learned_addr learned my addr 172.21.15.98:0/1374895065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430ed9d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310108c70 0x7f43101a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430e59c700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310073790 0x7f4310198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 -- 172.21.15.98:0/1374895065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310073790 msgr2=0x7f4310198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310073790 0x7f4310198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 -- 172.21.15.98:0/1374895065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310108c70 msgr2=0x7f43101a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310108c70 0x7f43101a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 -- 172.21.15.98:0/1374895065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43000095f0 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f430dd9b700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4300004710 tx=0x7f4300009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430001d070 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f430000be70 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300021410 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.830+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43101a0e90 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.831+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43101a1200 con 0x7f4310074d70 2024-03-18T20:32:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.831+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310107a80 con 0x7f4310074d70 2024-03-18T20:32:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.833+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f430000f6f0 con 0x7f4310074d70 2024-03-18T20:32:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.834+0000 7f42ff7fe700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 0x7f42f80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.834+0000 7f430e59c700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 0x7f42f80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.834+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f43000ce2e0 con 0x7f4310074d70 2024-03-18T20:32:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.834+0000 7f430e59c700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 0x7f42f80a35e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f431019a270 tx=0x7f430400b5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:32.837+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4300097740 con 0x7f4310074d70 2024-03-18T20:32:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:32 smithi169 ceph-mon[48529]: pgmap v462: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 552 op/s 2024-03-18T20:32:32.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:32 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4117691820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:32:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:32 smithi136 ceph-mon[48073]: pgmap v462: 129 pgs: 129 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 552 op/s 2024-03-18T20:32:32.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:32 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4117691820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:32:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.055+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f431002d0f0 con 0x7f4310074d70 2024-03-18T20:32:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.055+0000 7f42ff7fe700 1 -- 172.21.15.98:0/1374895065 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f430009ad60 con 0x7f4310074d70 2024-03-18T20:32:33.059 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:32:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 msgr2=0x7f42f80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 0x7f42f80a35e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f431019a270 tx=0x7f430400b5d0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 msgr2=0x7f4310199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4300004710 tx=0x7f4300009580 comp rx=0 tx=0).stop 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 shutdown_connections 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f42f80a1120 0x7f42f80a35e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4310108c70 0x7f43101a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4310073790 0x7f4310198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 --2- 172.21.15.98:0/1374895065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4310074d70 0x7f4310199330 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 >> 172.21.15.98:0/1374895065 conn(0x7f43100fc2f0 msgr2=0x7f4310104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.058+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 shutdown_connections 2024-03-18T20:32:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.059+0000 7f4314d5d700 1 -- 172.21.15.98:0/1374895065 wait complete. 2024-03-18T20:32:33.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.271+0000 7f06c9ffd700 1 -- 172.21.15.98:0/2985291318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 msgr2=0x7f06c40ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.271+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/2985291318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40ff360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009990 tx=0x7f06b4009ca0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 -- 172.21.15.98:0/2985291318 shutdown_connections 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/2985291318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c4100660 0x7f06c4104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/2985291318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40ff360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/2985291318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c40ffca0 0x7f06c4100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 -- 172.21.15.98:0/2985291318 >> 172.21.15.98:0/2985291318 conn(0x7f06c40fa870 msgr2=0x7f06c40fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 -- 172.21.15.98:0/2985291318 shutdown_connections 2024-03-18T20:32:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 -- 172.21.15.98:0/2985291318 wait complete. 2024-03-18T20:32:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.272+0000 7f06c9ffd700 1 Processor -- start 2024-03-18T20:32:33.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.274+0000 7f06c9ffd700 1 -- start start 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.274+0000 7f06c9ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c9ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c40ffca0 0x7f06c4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c9ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c9ffd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40722e0 con 0x7f06c40ffca0 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c9ffd700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4072420 con 0x7f06c40fef40 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c9ffd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40725b0 con 0x7f06c4100660 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.98:45832/0 (socket says 172.21.15.98:45832) 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 -- 172.21.15.98:0/4067738066 learned_addr learned my addr 172.21.15.98:0/4067738066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c2ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c40ffca0 0x7f06c4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 -- 172.21.15.98:0/4067738066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 msgr2=0x7f06c40707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 -- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c40ffca0 msgr2=0x7f06c4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c40ffca0 0x7f06c4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.275+0000 7f06c3fff700 1 -- 172.21.15.98:0/4067738066 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40095f0 con 0x7f06c4100660 2024-03-18T20:32:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.277+0000 7f06c3fff700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00b970 tx=0x7f06bc00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.278+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc00d860 con 0x7f06c4100660 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.278+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c4072890 con 0x7f06c4100660 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.278+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c4072df0 con 0x7f06c4100660 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.279+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f06bc00de50 con 0x7f06c4100660 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.279+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc017400 con 0x7f06c4100660 2024-03-18T20:32:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.280+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b0005320 con 0x7f06c4100660 2024-03-18T20:32:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.284+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f06bc00d9c0 con 0x7f06c4100660 2024-03-18T20:32:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.285+0000 7f06c0ff9700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 0x7f06ac0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.285+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f06bc006e40 con 0x7f06c4100660 2024-03-18T20:32:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.285+0000 7f06c37fe700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 0x7f06ac0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.286+0000 7f06c37fe700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 0x7f06ac0a36d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003780 tx=0x7f06b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.289+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06bc095e80 con 0x7f06c4100660 2024-03-18T20:32:33.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.497+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06b0005f70 con 0x7f06c4100660 2024-03-18T20:32:33.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.497+0000 7f06c0ff9700 1 -- 172.21.15.98:0/4067738066 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f06bc0994a0 con 0x7f06c4100660 2024-03-18T20:32:33.501 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:32:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 msgr2=0x7f06ac0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 0x7f06ac0a36d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f06b4003780 tx=0x7f06b4005fb0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 msgr2=0x7f06c4071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00b970 tx=0x7f06bc00bc80 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 shutdown_connections 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c4100660 0x7f06c4071d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f06ac0a1210 0x7f06ac0a36d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c40fef40 0x7f06c40707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 --2- 172.21.15.98:0/4067738066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06c40ffca0 0x7f06c4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 >> 172.21.15.98:0/4067738066 conn(0x7f06c40fa870 msgr2=0x7f06c4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 shutdown_connections 2024-03-18T20:32:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.500+0000 7f06c9ffd700 1 -- 172.21.15.98:0/4067738066 wait complete. 2024-03-18T20:32:33.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/4201665993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b740ff020 msgr2=0x7f4b740ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/4201665993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b740ff020 0x7f4b740ff440 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009990 tx=0x7f4b64009ca0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/4201665993 shutdown_connections 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/4201665993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b74102990 0x7f4b74109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/4201665993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b740ff020 0x7f4b740ff440 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/4201665993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 0x7f4b74102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/4201665993 >> 172.21.15.98:0/4201665993 conn(0x7f4b740fa950 msgr2=0x7f4b740fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/4201665993 shutdown_connections 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.638+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/4201665993 wait complete. 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 Processor -- start 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 -- start start 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 0x7f4b74101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 0x7f4b740fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74100c80 con 0x7f4b74102990 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74100df0 con 0x7f4b740ff020 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b7a8d2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741a0cb0 con 0x7f4b74101ff0 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47674/0 (socket says 172.21.15.98:47674) 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 -- 172.21.15.98:0/230729876 learned_addr learned my addr 172.21.15.98:0/230729876 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b737fe700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 0x7f4b740fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b73fff700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 0x7f4b74101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 -- 172.21.15.98:0/230729876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 msgr2=0x7f4b740fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 0x7f4b740fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 -- 172.21.15.98:0/230729876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 msgr2=0x7f4b74101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.639+0000 7f4b78e6f700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 0x7f4b74101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b78e6f700 1 -- 172.21.15.98:0/230729876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640095f0 con 0x7f4b74102990 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b73fff700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 0x7f4b74101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:32:33.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b78e6f700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007ee0 tx=0x7f4b6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c00dd70 con 0x7f4b74102990 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b741a0f40 con 0x7f4b74102990 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4b6c00f040 con 0x7f4b74102990 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c004790 con 0x7f4b74102990 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.640+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b741a13c0 con 0x7f4b74102990 2024-03-18T20:32:33.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.641+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b741083c0 con 0x7f4b74102990 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.642+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4b6c003a00 con 0x7f4b74102990 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.643+0000 7f4b717fa700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 0x7f4b5c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.643+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f4b6c0ccd80 con 0x7f4b74102990 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.643+0000 7f4b73fff700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 0x7f4b5c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.643+0000 7f4b73fff700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 0x7f4b5c0a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400b5c0 tx=0x7f4b64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.647+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4b6c096150 con 0x7f4b74102990 2024-03-18T20:32:33.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/401738604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1374895065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:33 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4067738066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.841+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4b7402d0f0 con 0x7f4b74102990 2024-03-18T20:32:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.841+0000 7f4b717fa700 1 -- 172.21.15.98:0/230729876 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4b6c0230a0 con 0x7f4b74102990 2024-03-18T20:32:33.844 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 msgr2=0x7f4b5c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 0x7f4b5c0a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400b5c0 tx=0x7f4b64005fd0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 msgr2=0x7f4b741004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007ee0 tx=0x7f4b6c00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 shutdown_connections 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b74102990 0x7f4b741004e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f4b5c0a1120 0x7f4b5c0a35e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b740ff020 0x7f4b74101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.843+0000 7f4b7a8d2700 1 --2- 172.21.15.98:0/230729876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b74101ff0 0x7f4b740fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.844+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 >> 172.21.15.98:0/230729876 conn(0x7f4b740fa950 msgr2=0x7f4b74105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.844+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 shutdown_connections 2024-03-18T20:32:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.844+0000 7f4b7a8d2700 1 -- 172.21.15.98:0/230729876 wait complete. 2024-03-18T20:32:33.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T20:32:33.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/401738604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1374895065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:33 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4067738066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- 172.21.15.98:0/1781093881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 msgr2=0x7f73c8108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/1781093881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c8108090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b0a0 tx=0x7f73c000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- 172.21.15.98:0/1781093881 shutdown_connections 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/1781093881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c8108090 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/1781093881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c8101100 0x7f73c8101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/1781093881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c80ffe30 0x7f73c8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- 172.21.15.98:0/1781093881 >> 172.21.15.98:0/1781093881 conn(0x7f73c80fa880 msgr2=0x7f73c80fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- 172.21.15.98:0/1781093881 shutdown_connections 2024-03-18T20:32:33.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- 172.21.15.98:0/1781093881 wait complete. 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 Processor -- start 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.978+0000 7f73ce0e1700 1 -- start start 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c80ffe30 0x7f73c8196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c8101100 0x7f73c819e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c819e790 con 0x7f73c81007d0 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c819e900 con 0x7f73c80ffe30 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c819eac0 con 0x7f73c8101100 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47696/0 (socket says 172.21.15.98:47696) 2024-03-18T20:32:33.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 -- 172.21.15.98:0/599094771 learned_addr learned my addr 172.21.15.98:0/599094771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c7fff700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c8101100 0x7f73c819e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c77fe700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c80ffe30 0x7f73c8196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 -- 172.21.15.98:0/599094771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c8101100 msgr2=0x7f73c819e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c8101100 0x7f73c819e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 -- 172.21.15.98:0/599094771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c80ffe30 msgr2=0x7f73c8196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c80ffe30 0x7f73c8196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 -- 172.21.15.98:0/599094771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0009d00 con 0x7f73c81007d0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c6ffd700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c920 tx=0x7f73b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8007b00 con 0x7f73c81007d0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f73b800ce70 con 0x7f73c81007d0 2024-03-18T20:32:33.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c819eda0 con 0x7f73c81007d0 2024-03-18T20:32:33.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8005440 con 0x7f73c81007d0 2024-03-18T20:32:33.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.979+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c819f2f0 con 0x7f73c81007d0 2024-03-18T20:32:33.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.980+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8067180 con 0x7f73c81007d0 2024-03-18T20:32:33.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.981+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f73b80055a0 con 0x7f73c81007d0 2024-03-18T20:32:33.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.982+0000 7f73c4ff9700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 0x7f73b00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:33.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.982+0000 7f73c77fe700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 0x7f73b00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:33.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.982+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f73b80cc7f0 con 0x7f73c81007d0 2024-03-18T20:32:33.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.982+0000 7f73c77fe700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 0x7f73b00a3660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f73bc006fd0 tx=0x7f73bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:33.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:33.986+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73b8095bc0 con 0x7f73c81007d0 2024-03-18T20:32:33.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/401738604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1374895065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:32:33.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:33 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4067738066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:32:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.319+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f73c819f630 con 0x7f73c81007d0 2024-03-18T20:32:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.327+0000 7f73c4ff9700 1 -- 172.21.15.98:0/599094771 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+733394 (secure 0 0 0) 0x7f73b801d090 con 0x7f73c81007d0 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 18, 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 14152, 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-18T20:32:34.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3114713517 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3114713517 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/3114713517", 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-03-18T20:19:45.512657+0000", 2024-03-18T20:32:34.342 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540138322906710015, 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24109, 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540138322906710015, 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-18T20:32:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-18T20:32:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-18T20:32:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T20:32:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-18T20:32:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-18T20:32:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T20:32:34.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-18T20:32:34.378 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap" 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-18T20:32:34.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-18T20:32:34.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-18T20:32:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T20:32:34.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-18T20:32:34.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-18T20:32:34.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-18T20:32:34.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T20:32:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-18T20:32:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-18T20:32:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T20:32:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-18T20:32:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-18T20:32:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-18T20:32:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.428 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.429 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T20:32:34.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-18T20:32:34.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-18T20:32:34.437 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-18T20:32:34.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.445 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.446 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.456 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-18T20:32:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-18T20:32:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T20:32:34.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T20:32:34.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T20:32:34.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T20:32:34.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.476 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T20:32:34.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-18T20:32:34.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T20:32:34.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T20:32:34.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.502 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-18T20:32:34.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:34.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.515 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.529 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:34.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-18T20:32:34.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.544 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.547 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-18T20:32:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-18T20:32:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-18T20:32:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T20:32:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-18T20:32:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T20:32:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.597 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-18T20:32:34.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.600 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.601 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.609 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.612 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.615 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.617 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.618 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.619 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.623 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.635 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.641 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.644 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:34.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.649 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.661 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-18T20:32:34.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T20:32:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-18T20:32:34.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:34.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:34.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.675 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-18T20:32:34.680 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.683 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:34.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:34.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.694 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.697 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.699 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-18T20:32:34.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.704 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.708 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.714 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-18T20:32:34.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-18T20:32:34.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.724 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.727 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.735 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.737 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.738 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.741 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.744 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.748 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.749 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-18T20:32:34.750 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T20:32:34.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-18T20:32:34.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.762 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.763 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.764 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.767 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.768 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-18T20:32:34.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-18T20:32:34.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.772 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-18T20:32:34.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-18T20:32:34.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T20:32:34.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.786 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.792 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.794 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24116, 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540138322906710015, 2024-03-18T20:32:34.797 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.799 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-18T20:32:34.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-18T20:32:34.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-18T20:32:34.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-18T20:32:34.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-18T20:32:34.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-18T20:32:34.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.818 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.820 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.821 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.822 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-18T20:32:34.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:34.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap" 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-18T20:32:34.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-18T20:32:34.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-18T20:32:34.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-18T20:32:34.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.835 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.836 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.837 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.838 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-18T20:32:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-18T20:32:34.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.853 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.854 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.857 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-18T20:32:34.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:34.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.865 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.871 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-18T20:32:34.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:34.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.888 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T20:32:34.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T20:32:34.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T20:32:34.892 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.897 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-18T20:32:34.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-18T20:32:34.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.902 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.903 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T20:32:34.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-18T20:32:34.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T20:32:34.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T20:32:34.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T20:32:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T20:32:34.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.910 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.911 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.912 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.913 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.914 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.919 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.921 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.925 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.929 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.931 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.934 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.942 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.944 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.954 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.955 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:34.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-18T20:32:34.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.964 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-18T20:32:34.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.971 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.972 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.978 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-18T20:32:34.980 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.981 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.985 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.986 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-18T20:32:34.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-18T20:32:34.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:34.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:34.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.995 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.997 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:34.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.001 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.002 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.004 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.007 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.008 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.010 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.012 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.019 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.022 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.028 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-18T20:32:35.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-18T20:32:35.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-18T20:32:35.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-18T20:32:35.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.038 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.039 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.042 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.045 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.050 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:35.052 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.054 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.058 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-18T20:32:35.059 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.060 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.061 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.064 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.069 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.071 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.072 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.073 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.077 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.080 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.083 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.084 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.086 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T20:32:35.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.091 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.104 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.105 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-18T20:32:35.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:35.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:35.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:35.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.123 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.129 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T20:32:35.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.140 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-18T20:32:35.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-18T20:32:35.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.144 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.145 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.146 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.147 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-18T20:32:35.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.149 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.150 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-18T20:32:35.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-18T20:32:35.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-18T20:32:35.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.157 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.163 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.168 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-18T20:32:35.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.174 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.175 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.176 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-18T20:32:35.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.195 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.197 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.203 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.206 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.211 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-18T20:32:35.216 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.220 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-03-18T20:32:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "restful" 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-18T20:32:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-18T20:32:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T20:32:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T20:32:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T20:32:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T20:32:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-18T20:32:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T20:32:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T20:32:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.252 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.255 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap" 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-18T20:32:35.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.262 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-18T20:32:35.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:35.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-18T20:32:35.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-18T20:32:35.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.273 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.284 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.299 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.328 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T20:32:35.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T20:32:35.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T20:32:35.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T20:32:35.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T20:32:35.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T20:32:35.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T20:32:35.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.363 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T20:32:35.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T20:32:35.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T20:32:35.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T20:32:35.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T20:32:35.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T20:32:35.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.370 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.371 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.372 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.375 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.383 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.389 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.390 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-18T20:32:35.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:35.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.401 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-18T20:32:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-18T20:32:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-18T20:32:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-18T20:32:35.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-18T20:32:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-18T20:32:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.433 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.443 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.445 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.448 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-18T20:32:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T20:32:35.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.485 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-18T20:32:35.490 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.491 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.492 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.498 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.499 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.500 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.502 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.515 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.519 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-18T20:32:35.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.521 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.522 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.524 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.535 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.538 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.543 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.546 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.550 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-18T20:32:35.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.553 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.561 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-18T20:32:35.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.565 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.567 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.569 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.570 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.571 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.572 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.573 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.577 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-18T20:32:35.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-18T20:32:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-18T20:32:35.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-18T20:32:35.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-18T20:32:35.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-18T20:32:35.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-18T20:32:35.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-18T20:32:35.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-18T20:32:35.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-18T20:32:35.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.657 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-18T20:32:35.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-18T20:32:35.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-18T20:32:35.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.663 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.671 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-18T20:32:35.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-18T20:32:35.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-18T20:32:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.680 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-18T20:32:35.687 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-18T20:32:35.688 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-18T20:32:35.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-18T20:32:35.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-18T20:32:35.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 3, 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3930321953 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-03-18T20:32:35.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3650538312 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1714041692 2024-03-18T20:32:35.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1193712487 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-18T20:32:35.694 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:34 smithi136 ceph-mon[48073]: pgmap v463: 129 pgs: 129 active+clean; 179 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 528 op/s 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:34 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/230729876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:34 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/599094771' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:34 smithi169 ceph-mon[48529]: pgmap v463: 129 pgs: 129 active+clean; 179 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 528 op/s 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:34 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/230729876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:32:35.696 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:34 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/599094771' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T20:32:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 msgr2=0x7f73b00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 0x7f73b00a3660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f73bc006fd0 tx=0x7f73bc008040 comp rx=0 tx=0).stop 2024-03-18T20:32:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 msgr2=0x7f73c81971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c920 tx=0x7f73b800cc30 comp rx=0 tx=0).stop 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 shutdown_connections 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f73c8101100 0x7f73c819e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f73b00a11a0 0x7f73b00a3660 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73c80ffe30 0x7f73c8196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 --2- 172.21.15.98:0/599094771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73c81007d0 0x7f73c81971e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 >> 172.21.15.98:0/599094771 conn(0x7f73c80fa880 msgr2=0x7f73c80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 shutdown_connections 2024-03-18T20:32:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:34.332+0000 7f73ce0e1700 1 -- 172.21.15.98:0/599094771 wait complete. 2024-03-18T20:32:35.704 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T20:32:35.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T20:32:35.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:34 smithi098 ceph-mon[43462]: pgmap v463: 129 pgs: 129 active+clean; 179 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 528 op/s 2024-03-18T20:32:35.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:34 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/230729876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:32:35.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:34 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/599094771' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T20:32:35.946 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:32:36.883 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:32:36.883 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001279227, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000175997, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:36.884 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 5, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 102, 2024-03-18T20:32:36.885 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3168, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 94064, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.006350604, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.004285767, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001358111, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 101, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.004045893, 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000040058 2024-03-18T20:32:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3168, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 94064 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 134, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 131, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 447680, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 74208, 2024-03-18T20:32:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018313671, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006943076, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.010004877, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.002585621, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 130, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.004886554, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000037588 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 447680, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 74208 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-18T20:32:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148863, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 28, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-18T20:32:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 135, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 428762, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 135, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 135, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 428762, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:36.895 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:32:37.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:36 smithi098 ceph-mon[43462]: pgmap v464: 129 pgs: 129 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 518 op/s 2024-03-18T20:32:37.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:36 smithi169 ceph-mon[48529]: pgmap v464: 129 pgs: 129 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 518 op/s 2024-03-18T20:32:37.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:36 smithi136 ceph-mon[48073]: pgmap v464: 129 pgs: 129 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 518 op/s 2024-03-18T20:32:37.246 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-18T20:32:37.247 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-18T20:32:37.247 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-18T20:32:37.247 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-18T20:32:37.247 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-18T20:32:37.247 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T20:32:37.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T20:32:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-18T20:32:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001208623, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000258312, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 5, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 105, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3168, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 94592, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:38.510 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.007488905, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.005171730, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001552114, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 104, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.004264973, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000041009 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:38.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3168, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 94592 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 139, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 136, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 449136, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 76880, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018488576, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006477330, 2024-03-18T20:32:38.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.010992471, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.002672017, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 135, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.006721574, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000049789 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 449136, 2024-03-18T20:32:38.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 76880 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:38.514 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:38.515 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-18T20:32:38.516 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148823, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 27, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:38.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 140, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 429515, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 140, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 140, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 429515, 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-18T20:32:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:38.520 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:38.520 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:38.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:38.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:38.520 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:32:39.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:38 smithi098 ceph-mon[43462]: pgmap v465: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 853 KiB/s wr, 456 op/s 2024-03-18T20:32:39.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:38 smithi169 ceph-mon[48529]: pgmap v465: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 853 KiB/s wr, 456 op/s 2024-03-18T20:32:39.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:38 smithi136 ceph-mon[48073]: pgmap v465: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 853 KiB/s wr, 456 op/s 2024-03-18T20:32:39.704 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T20:32:39.705 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T20:32:39.705 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-18T20:32:39.705 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-18T20:32:39.705 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-18T20:32:39.705 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T20:32:39.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T20:32:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:32:40.890 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:40 smithi098 ceph-mon[43462]: pgmap v466: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 567 KiB/s wr, 378 op/s 2024-03-18T20:32:41.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:40 smithi169 ceph-mon[48529]: pgmap v466: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 567 KiB/s wr, 378 op/s 2024-03-18T20:32:41.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:40 smithi136 ceph-mon[48073]: pgmap v466: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 567 KiB/s wr, 378 op/s 2024-03-18T20:32:41.608 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:32:41.608 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:41.608 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-18T20:32:41.608 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-18T20:32:41.608 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001336454, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000336039, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:41.609 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 101, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 160, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 93792, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:41.610 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.006564615, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.004685783, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001059332, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 99, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003443962, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000034787 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:41.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 93792 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 140, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 135, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 451232, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 75840, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018839259, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.007233555, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.010417720, 2024-03-18T20:32:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.002551284, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 135, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.006907571, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000051167 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 451232, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 75840 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148823, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 27, 2024-03-18T20:32:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-18T20:32:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 137, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 429022, 2024-03-18T20:32:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 137, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 137, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 429022, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:41.619 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:32:41.989 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-18T20:32:41.989 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-18T20:32:41.989 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-18T20:32:41.989 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-18T20:32:41.989 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-18T20:32:41.989 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T20:32:41.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T20:32:42.214 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T20:32:42.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:42 smithi098 ceph-mon[43462]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 322 op/s 2024-03-18T20:32:42.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:42 smithi169 ceph-mon[48529]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 322 op/s 2024-03-18T20:32:42.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:42 smithi136 ceph-mon[48073]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 322 op/s 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 144, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 139, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 452416, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 77920, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:43.223 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.020506944, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.007250873, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.011569114, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.002687538, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 139, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.010206860, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000073430 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.224 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 452416, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 77920 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 104, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 160, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 94320, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.006859842, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.005061590, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000962715, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 102, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.003272615, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032084 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 94320 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000172877, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000116080, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:43.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.228 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:43.229 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:43.230 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148923, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-18T20:32:43.231 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:43.232 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 141, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 429645, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 141, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 141, 2024-03-18T20:32:43.233 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 429645, 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-18T20:32:43.234 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-18T20:32:43.579 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T20:32:43.580 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T20:32:43.580 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-18T20:32:43.580 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-18T20:32:43.580 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-18T20:32:43.580 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T20:32:43.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T20:32:43.829 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 12093, 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 11978, 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 3203111, 2024-03-18T20:32:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 96212068, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 11, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.846950733, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.294287454, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.164026239, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.348239113, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 11974, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 6.242353902, 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000521325 2024-03-18T20:32:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 262, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000042250, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000161 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 82064, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 35744 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 13407, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 15646, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 4944784, 2024-03-18T20:32:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 102528103, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 7, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 5, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 1.129137744, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.384454754, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.253853194, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.463798985, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 15648, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 5.007943952, 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000320037 2024-03-18T20:32:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 6583, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.001193804, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000181 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 495040, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 83696 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 601008, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 136939, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 209573169, 2024-03-18T20:32:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 87651375, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 15, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 16.408991451, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 6.433855127, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 14.669035590, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.244404604, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 136937, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 7.426063722, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000054229 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 705264, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.073386509, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000104 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 257920 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2024-03-18T20:32:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 5035, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 26.604142904, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.005283841 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 17658, 2024-03-18T20:32:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 2.393837889, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000135566 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 49463, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 49463, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 49463, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 359.456303601, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.007267175 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 0, 2024-03-18T20:32:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 421, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 33, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 32, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 8, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 24, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 20, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 0, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 2, 2024-03-18T20:32:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 2, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 2, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 60258, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 60090, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2024-03-18T20:32:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 4, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 234, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 13000, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 24762, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 6, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 541308, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 10534, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 319, 2024-03-18T20:32:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 57559 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 455, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 0, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 0, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 0, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 547, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 3, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 10395, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 10392, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 33, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 133558, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 25734, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 25734, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 107824, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 4395, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 158, 2024-03-18T20:32:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 30, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 30, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 133, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 5, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 58193822, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 214413611, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 0, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2461, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 14.223589420, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.005779597 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 0 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 26, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 10418, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 10392, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 21, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 329, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 308, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 31, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 23461, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 23430, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 2, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 10409, 2024-03-18T20:32:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 10407, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 96428, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 207156 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 53062, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 1, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 49463, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 17, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 2, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 10349, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 18.030280787, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001742224 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 642, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 2.796118062, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.004355324 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.006634699, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002211566 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 53, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.114482686, 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002160050 2024-03-18T20:32:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 21062, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 21.131716266, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001003310 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 496, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 1.349708418, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002721186 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 9957, 2024-03-18T20:32:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 302.764286494, 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.030407179 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 47, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 4.238107563, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.090172501 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.001750474, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000875237 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2024-03-18T20:32:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 9, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.022480037, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002497781 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.002781839, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000927279 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 6840, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 8.997956276, 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001315490 2024-03-18T20:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 3, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 2, 2024-03-18T20:32:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 1, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 16446, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 16446, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 396, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 312771, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 118, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 178, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 116504, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 1295, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2024-03-18T20:32:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 30160, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 220342813, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 37, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 30160, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 30160, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 557.153697620, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.018473265 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 30160, 2024-03-18T20:32:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 36955 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 30160, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 626, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 29534, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 98, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 3988, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 626, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 3087, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 347, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2024-03-18T20:32:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 33, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 21575, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 6009, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2024-03-18T20:32:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 6, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2024-03-18T20:32:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 89, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 9, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 12, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 12, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 979, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 0, 2024-03-18T20:32:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 207 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 1, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 49497 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 23, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 9, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 10392, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 626504, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 161314068, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 626504, 2024-03-18T20:32:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 626504, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 161314068, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 30160, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 221475429, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 30160, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 221475429, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 30160, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 30160, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 30160, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 30160, 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:44.878 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 10392, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1132616, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 10392, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:44.879 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 626, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1132616, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x55984b16c0a0": { 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 133558, 2024-03-18T20:32:44.880 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 210219307, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 133558, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 2461, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 210219307, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:44.881 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-18T20:32:44.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:44 smithi136 ceph-mon[48073]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 218 op/s 2024-03-18T20:32:45.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:44 smithi098 ceph-mon[43462]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 218 op/s 2024-03-18T20:32:45.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:44 smithi169 ceph-mon[48529]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 218 op/s 2024-03-18T20:32:45.186 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-18T20:32:45.186 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-18T20:32:45.186 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=455 2024-03-18T20:32:45.186 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-18T20:32:45.187 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=33 2024-03-18T20:32:45.187 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T20:32:45.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T20:32:45.409 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:32:46.365 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-18T20:32:46.365 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:46.365 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 473, 2024-03-18T20:32:46.365 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 475, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 126069, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 147167, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 6, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.035720178, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.013302981, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.008533213, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.009392452, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 473, 2024-03-18T20:32:46.366 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.017094226, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000036140 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 405, 2024-03-18T20:32:46.367 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 508, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 99566, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 374737, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 5, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 4, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.034233366, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.011726491, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.007545520, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.011246775, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 506, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.016973558, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000033544 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.368 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 249088 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 620, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 614, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 628368, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 224382, 2024-03-18T20:32:46.369 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.063444792, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.019320210, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.023228465, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.018074018, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 614, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.030615913, 2024-03-18T20:32:46.370 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000049863 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 471792, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 79456 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-03-18T20:32:46.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2247, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.054522928, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000024264 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.372 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 0, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 0, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2024-03-18T20:32:46.373 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 0, 2024-03-18T20:32:46.374 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 1, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2024-03-18T20:32:46.375 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:46.376 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:46.377 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 0 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:46.378 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:46.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 4, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 1, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2024-03-18T20:32:46.380 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 0, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 4194304, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 4195872, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 4195731, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 4 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 3, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 3, 2024-03-18T20:32:46.381 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 1, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 1, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 0, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 46004, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 207156 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2024-03-18T20:32:46.382 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.383 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.384 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2024-03-18T20:32:46.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.387 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.388 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.391 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2024-03-18T20:32:46.392 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 2, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 1, 2024-03-18T20:32:46.394 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 2, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 3063, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:46.395 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:46.396 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-03-18T20:32:46.397 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 190454, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 52, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:46.398 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 178, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 8320, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 13, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-03-18T20:32:46.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 1343, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 21, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 3, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 1343, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1343, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.582987112, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000434093 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2024-03-18T20:32:46.400 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1343, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 1344 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 1343, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 1343, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 0, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 868, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 0, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 473, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 0, 2024-03-18T20:32:46.401 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 1, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:46.402 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2024-03-18T20:32:46.403 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 89, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 4, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 5, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 5, 2024-03-18T20:32:46.404 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 2, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 0 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 0 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.405 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 1494, 2024-03-18T20:32:46.406 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 708838, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 1494, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 1494, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 708838, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:46.407 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 1343, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 1573, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 1343, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1573, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 1343, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 1343, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 1343, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1343, 2024-03-18T20:32:46.409 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:46.410 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x55e84e4505a0": { 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:46.411 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:46.412 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:46.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:46.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:46.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:46.414 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-18T20:32:46.737 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T20:32:46.737 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T20:32:46.737 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-18T20:32:46.737 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-18T20:32:46.738 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-18T20:32:46.738 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T20:32:46.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T20:32:46.959 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:32:46.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:46 smithi136 ceph-mon[48073]: pgmap v469: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 184 KiB/s wr, 129 op/s 2024-03-18T20:32:47.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:46 smithi098 ceph-mon[43462]: pgmap v469: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 184 KiB/s wr, 129 op/s 2024-03-18T20:32:47.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:46 smithi169 ceph-mon[48529]: pgmap v469: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 184 KiB/s wr, 129 op/s 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 29, 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 31, 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 7670, 2024-03-18T20:32:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 12387, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 9, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.005303889, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000515651, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.002866323, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000505309, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 29, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.015047078, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000518864 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 15, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000002524, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000168 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2080, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 1696 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 201, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 195, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 482860, 2024-03-18T20:32:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 133192, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 10, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 7, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.024889126, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.007660832, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.015982858, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.003248995, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 194, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.017538420, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000090404 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 18, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000003209, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000178 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 477296, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 82848 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 4545, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 2389, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 2051581, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 727510, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 6, 2024-03-18T20:32:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 4, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.200731633, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.086992327, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.152883599, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.002031795, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2388, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.057577634, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000024111 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 6722, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000953774, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000141 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 3168, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 249360 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 47, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.001465673, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000031184 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000014296, 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000007148 2024-03-18T20:32:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 3, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 1, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000281067, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000281067 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 2, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 11, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 25, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 7, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 18, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 13, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 92, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 0, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 3, 2024-03-18T20:32:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 276, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 229, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 45, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 4, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 234, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 0, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 2 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2024-03-18T20:32:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 499, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 272, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 273, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2024-03-18T20:32:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2024-03-18T20:32:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 4, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 4, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 1, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 1, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 0, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 4194304, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 4195872, 2024-03-18T20:32:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 0, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 4, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.006453879, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001613469 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 0 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 29, 2024-03-18T20:32:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 75, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 46, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 28, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 312, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 284, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 25, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 117, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 92, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 0, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 1, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 1, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 49572, 2024-03-18T20:32:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 198964 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 3, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 3, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 12, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 0, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000281067, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000281067 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 2, 2024-03-18T20:32:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 1, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 1, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 1, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 1, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 399, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 238823, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 79, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 178, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 116669, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 423, 2024-03-18T20:32:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 40, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 11870, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 27, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 40, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 40, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.200995576, 2024-03-18T20:32:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.005024889 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 40, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 62 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 40, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 0, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 40, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2024-03-18T20:32:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 10, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 4, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 7, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 0, 2024-03-18T20:32:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 6, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 20, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2024-03-18T20:32:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2024-03-18T20:32:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2024-03-18T20:32:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 89, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 4, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 8, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 8, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 14, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 0, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 1 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2024-03-18T20:32:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 1, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 3 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 0, 2024-03-18T20:32:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 0, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 4771, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 2100475, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 4771, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4771, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 2100475, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 40, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 11870, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 40, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 11870, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 40, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 40, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 40, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 40, 2024-03-18T20:32:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x5600251720a0": { 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 4, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1568, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 4, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4, 2024-03-18T20:32:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1568, 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:48.035 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-18T20:32:48.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:47 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:32:48.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:47 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:32:48.182 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:47 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T20:32:48.322 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-18T20:32:48.323 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-18T20:32:48.323 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-18T20:32:48.323 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=499 2024-03-18T20:32:48.323 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-18T20:32:48.323 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T20:32:48.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T20:32:48.546 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T20:32:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:48 smithi136 ceph-mon[48073]: pgmap v470: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 50 op/s 2024-03-18T20:32:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:32:48.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:32:48.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:48 smithi136 ceph-mon[48073]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:32:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:48 smithi098 ceph-mon[43462]: pgmap v470: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 50 op/s 2024-03-18T20:32:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:32:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:32:49.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:48 smithi098 ceph-mon[43462]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:32:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:48 smithi169 ceph-mon[48529]: pgmap v470: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 50 op/s 2024-03-18T20:32:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T20:32:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T20:32:49.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:48 smithi169 ceph-mon[48529]: from='mgr.14152 172.21.15.98:0/4131248672' entity='mgr.x' 2024-03-18T20:32:49.540 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-18T20:32:49.540 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:49.540 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 4, 2024-03-18T20:32:49.540 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 4, 2024-03-18T20:32:49.540 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 3008, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 448, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.001855747, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000055561, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000783336, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000878720, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000292906 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.541 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 3008, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 142, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 139, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 449952, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 78656, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:49.542 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.020564451, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.007178684, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.011543678, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.002951946, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 138, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.006913799, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000050099 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:49.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 449952, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 78656 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 106, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 160, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 94672, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.005930783, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.003927256, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000861147, 2024-03-18T20:32:49.544 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 104, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.005572346, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000053580 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 94672 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-03-18T20:32:49.545 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:49.546 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:49.547 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 148967, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 29, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:49.548 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 2, 2024-03-18T20:32:49.549 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 143, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 429905, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 143, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-03-18T20:32:49.550 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 143, 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 429905, 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-03-18T20:32:49.551 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-18T20:32:49.895 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-18T20:32:49.895 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-18T20:32:49.895 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-18T20:32:49.895 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-18T20:32:49.895 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-18T20:32:49.895 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T20:32:49.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T20:32:50.144 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:32:50.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:50 smithi169 ceph-mon[48529]: pgmap v471: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 14 op/s 2024-03-18T20:32:51.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:50 smithi098 ceph-mon[43462]: pgmap v471: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 14 op/s 2024-03-18T20:32:51.162 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 967, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 967, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 71604280, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 299951, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 8, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 4, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.134366635, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.028322305, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.066414146, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.029525214, 2024-03-18T20:32:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 966, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.042822106, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000044329 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3008, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 701, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 697, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 97586144, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 250565, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 6, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 5, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.121888285, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.021708496, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.079070157, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.017122749, 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 696, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.036400338, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000052299 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 469216, 2024-03-18T20:32:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 79760 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 341, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 443, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 42029128, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 356086, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 6, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 5, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.055798893, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.014522634, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.027895318, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.009440558, 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 441, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.018921159, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000042905 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 250656 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-18T20:32:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3043, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.095874558, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000031506 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 0, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 0, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 10541, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 1366, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 9175, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 8589, 2024-03-18T20:32:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 2, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 0, 2024-03-18T20:32:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 4, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 234, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 0, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 0, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 0, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 0 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 0, 2024-03-18T20:32:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 0, 2024-03-18T20:32:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-03-18T20:32:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 133558, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 133, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-03-18T20:32:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 0, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 58193822, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 214413611, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 214411590, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 133558 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 26, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 10418, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 10392, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 21, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 329, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 308, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 10541, 2024-03-18T20:32:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 23461, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 12920, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 75012, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 207156 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 0, 2024-03-18T20:32:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-03-18T20:32:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 1, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 3, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 2, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 2, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 0, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 0, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 3047, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 238687, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 77, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-18T20:32:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 178, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 7504600, 2024-03-18T20:32:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 42824, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 1851, 2024-03-18T20:32:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 26, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 4, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 1851, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1851, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1.331972021, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000719595 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-03-18T20:32:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1851, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1852 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 1851, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 1851, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 0, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 1006, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 0, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 842, 2024-03-18T20:32:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 2, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 0, 2024-03-18T20:32:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-03-18T20:32:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-03-18T20:32:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 89, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 4, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 11, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 11, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-03-18T20:32:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 0, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 2, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 0 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 0, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-03-18T20:32:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 0 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 0, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 0, 2024-03-18T20:32:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 2005, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 211028299, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 2005, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 2005, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 211028299, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1851, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 210236392, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1851, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 210236392, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1851, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 1851, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1851, 2024-03-18T20:32:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1851, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x564005710640": { 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:51.219 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-18T20:32:51.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:50 smithi136 ceph-mon[48073]: pgmap v471: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 14 op/s 2024-03-18T20:32:51.526 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-18T20:32:51.526 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-18T20:32:51.526 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-18T20:32:51.526 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-18T20:32:51.526 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-18T20:32:51.526 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T20:32:51.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T20:32:51.750 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:32:52.786 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-18T20:32:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 521, 2024-03-18T20:32:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 523, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 172545, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 162038, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 6, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.060114609, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.017752421, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.016345864, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.019011171, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 521, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.029094959, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000055844 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 843, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 944, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 216871, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 511649, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 5, 2024-03-18T20:32:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 4, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.063509195, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.021711368, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.015558048, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.021041418, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 943, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.036856405, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000039084 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3168, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 251280 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 152, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 148, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 469324, 2024-03-18T20:32:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 79911, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.025026731, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.009101157, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.013085683, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.003550696, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 147, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.009086838, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000061815 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 469056, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 79600 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2301, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.063518262, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000027604 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 1, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 0, 2024-03-18T20:32:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T20:32:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 0 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 0, 2024-03-18T20:32:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 2, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 1, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-03-18T20:32:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 0, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 4194304, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 4195339, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 4194844, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 2 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 3, 2024-03-18T20:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 3, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 1, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 1, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 0, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 45468, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 207156 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-03-18T20:32:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 1, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 1, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 0, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 1, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 0, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 0, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 1759, 2024-03-18T20:32:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 189440, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 52, 2024-03-18T20:32:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 178, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-18T20:32:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 8320, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 13, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-03-18T20:32:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 1359, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 21, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 3, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 1359, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1359, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.781983612, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000575411 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 1359, 2024-03-18T20:32:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 1360 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 1359, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 1359, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 880, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 477, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-03-18T20:32:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 1, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-03-18T20:32:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-03-18T20:32:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 89, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 4, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 5, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 5, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-03-18T20:32:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 0, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 2, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 0 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 0, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 0 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-03-18T20:32:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 1512, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 711903, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 1512, 2024-03-18T20:32:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1512, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 711903, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:52.829 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1359, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 1040, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1359, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1040, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.830 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 1359, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 1359, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 1359, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1359, 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:52.831 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:52.832 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x5637df7e05a0": { 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:52.833 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:52.834 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-18T20:32:52.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:52 smithi169 ceph-mon[48529]: pgmap v472: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-03-18T20:32:52.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:52 smithi098 ceph-mon[43462]: pgmap v472: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-03-18T20:32:52.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:52 smithi136 ceph-mon[48073]: pgmap v472: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-03-18T20:32:55.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:54 smithi098 ceph-mon[43462]: pgmap v473: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:32:55.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:54 smithi169 ceph-mon[48529]: pgmap v473: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:32:55.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:54 smithi136 ceph-mon[48073]: pgmap v473: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-18T20:32:55.399 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T20:32:55.399 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T20:32:55.400 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-18T20:32:55.400 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-18T20:32:55.400 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-18T20:32:55.400 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T20:32:55.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T20:32:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:32:56.911 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 4567, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 2312, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 2056318, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 463488, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 8, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 3, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.234458983, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.088395327, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.206938493, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.003499486, 2024-03-18T20:32:56.912 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2311, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.072529275, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000031384 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 6774, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001178525, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000173 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3008, 2024-03-18T20:32:56.913 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 205, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 197, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 487188, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 128802, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 10, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 6, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.028951870, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.009593705, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.018287325, 2024-03-18T20:32:56.914 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.003833120, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 196, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.019596064, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000099979 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 27, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000005487, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000203 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 481152, 2024-03-18T20:32:56.915 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 83440 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 11, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 115, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 2925, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 264349, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 7, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 5, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.008220706, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.004401671, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.002418974, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000248128, 2024-03-18T20:32:56.916 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 113, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.011768916, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000104149 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 253968 2024-03-18T20:32:56.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 1, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-MDSRank": { 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 37, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001370430, 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000037038 2024-03-18T20:32:56.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "finisher-PurgeQueue": { 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "queue_len": 0, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "complete_latency": { 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000017547, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000008773 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "mds": { 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "request": 4, 2024-03-18T20:32:56.919 INFO:teuthology.orchestra.run.smithi169.stdout: "reply": 3, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "reply_latency": { 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001829327, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000609775 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_reply": 0, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "forward": 1, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_complete": 0, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_fetch_keys": 0, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_commit": 11, 2024-03-18T20:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_split": 0, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_merge": 0, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes": 25, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_top": 7, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_bottom": 18, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pin_tail": 0, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_pinned": 13, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_expired": 92, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "inodes_with_caps": 3, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "caps": 3, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "subtrees": 3, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse": 277, 2024-03-18T20:32:56.921 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_hit": 231, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_forward": 1, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_discover": 45, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_dir_fetch": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_remote_ino": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "traverse_lock": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "load_cent": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "q": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "exported": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "exported_inodes": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "imported": 0, 2024-03-18T20:32:56.922 INFO:teuthology.orchestra.run.smithi169.stdout: "imported_inodes": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_dir_fetch": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_backtrace_fetch": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "openino_peer_discover": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rfiles": 4, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rbytes": 234, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "root_rsnaps": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_set_tag": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_inotable_repaired": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_inodes": 0, 2024-03-18T20:32:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_file_inodes": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_inode_file_caps": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_revoke": 1, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_grant": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps": 1, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_caps_dirty": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_cap_release": 0, 2024-03-18T20:32:56.924 INFO:teuthology.orchestra.run.smithi169.stdout: "process_request_cap_release": 0 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_cache": { 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update": 0, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_update_receipt": 501, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_try_discover": 273, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_send_discover": 274, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "dir_handle_discover": 0, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays": 0, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_delayed": 0, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "num_strays_enqueuing": 0, 2024-03-18T20:32:56.925 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_created": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_enqueued": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_reintegrated": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "strays_migrated": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_processing": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_enqueued": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "num_recovering_prioritized": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_started": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "recovery_completed": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_exportdir": 0, 2024-03-18T20:32:56.926 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_flush": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragmentdir": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_fragstats": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "ireq_inodestats": 0 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_log": { 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "evadd": 2, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "evex": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "evtrm": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "ev": 2, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "evexg": 0, 2024-03-18T20:32:56.927 INFO:teuthology.orchestra.run.smithi169.stdout: "evexd": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "segadd": 1, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "segex": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "segtrm": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "seg": 1, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "segexg": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "segexd": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "expos": 4194304, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "wrpos": 4195339, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "rdpos": 0, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "jlat": { 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 2, 2024-03-18T20:32:56.928 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.003434458, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.001717229 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "replayed": 0 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_mem": { 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "ino": 29, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "ino+": 75, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "ino-": 46, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "dir": 28, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "dir+": 313, 2024-03-18T20:32:56.929 INFO:teuthology.orchestra.run.smithi169.stdout: "dir-": 285, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "dn": 25, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "dn+": 117, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "dn-": 92, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "cap": 3, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "cap+": 3, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "cap-": 0, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "rss": 48604, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "heap": 198964 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_server": { 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_client_request": 6, 2024-03-18T20:32:56.930 INFO:teuthology.orchestra.run.smithi169.stdout: "dispatch_server_request": 0, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_request": 4, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_client_session": 7, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "handle_peer_request": 1, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "req_create_latency": { 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getattr_latency": { 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3, 2024-03-18T20:32:56.931 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.001829327, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000609775 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getfilelock_latency": { 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "req_link_latency": { 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookup_latency": { 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookuphash_latency": { 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.933 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupino_latency": { 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupname_latency": { 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupparent_latency": { 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.934 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lookupsnap_latency": { 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "req_lssnap_latency": { 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mkdir_latency": { 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mknod_latency": { 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.936 INFO:teuthology.orchestra.run.smithi169.stdout: "req_mksnap_latency": { 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "req_open_latency": { 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "req_readdir_latency": { 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.937 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rename_latency": { 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "req_renamesnap_latency": { 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "req_snapdiff_latency": { 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmdir_latency": { 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.939 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmsnap_latency": { 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "req_rmxattr_latency": { 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setattr_latency": { 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setdirlayout_latency": { 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setfilelock_latency": { 2024-03-18T20:32:56.941 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setlayout_latency": { 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "req_setxattr_latency": { 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.942 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "req_symlink_latency": { 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "req_unlink_latency": { 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.943 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_revoke_eviction": 0, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "cap_acquisition_throttle": 0, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "req_getvxattr_latency": { 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_sessions": { 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "session_count": 1, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "session_add": 1, 2024-03-18T20:32:56.944 INFO:teuthology.orchestra.run.smithi169.stdout: "session_remove": 0, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_open": 1, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "sessions_stale": 0, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "total_load": 2, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "average_load": 2, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "avg_session_uptime": 411, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "mdthresh_evicted": 0 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:56.945 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:56.946 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:56.947 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:56.948 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 254987, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 86, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 9272, 2024-03-18T20:32:56.949 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 178, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 117813, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 434, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.950 INFO:teuthology.orchestra.run.smithi169.stdout: "objecter": { 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_active": 0, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_laggy": 0, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send": 38, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_send_bytes": 11466, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_resend": 27, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_reply": 38, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_latency": { 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 38, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.217995389, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.005736720 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.951 INFO:teuthology.orchestra.run.smithi169.stdout: "op_inflight": 0, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "oplen_avg": { 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 38, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 61 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "op": 38, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "op_r": 0, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "op_w": 38, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "op_rmw": 0, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "op_pg": 0, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_stat": 0, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_create": 10, 2024-03-18T20:32:56.952 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_read": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_write": 2, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writefull": 6, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_writesame": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_append": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_zero": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_truncate": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_delete": 6, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_mapext": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_sparse_read": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_clonerange": 0, 2024-03-18T20:32:56.953 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_getxattr": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_setxattr": 20, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_cmpxattr": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_rmxattr": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_resetxattrs": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_call": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_watch": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_notify": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_pgls_filter": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "osdop_other": 0, 2024-03-18T20:32:56.954 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_active": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_send": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_resend": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "linger_ping": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_active": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_send": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolop_resend": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_active": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_send": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "poolstat_resend": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_active": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_send": 0, 2024-03-18T20:32:56.955 INFO:teuthology.orchestra.run.smithi169.stdout: "statfs_resend": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "command_active": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "command_send": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "command_resend": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "map_epoch": 89, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "map_full": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "map_inc": 4, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_sessions": 10, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_open": 10, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_session_close": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_laggy": 0, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_wr": 16, 2024-03-18T20:32:56.956 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_rd": 0, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_del": 1 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "oft": { 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_objs": 1, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_kv_pairs": 0, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_updates": 0, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "omap_total_removes": 0 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "osd-slow-ops": { 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "slow_ops_count": 4 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.957 INFO:teuthology.orchestra.run.smithi169.stdout: "purge_queue": { 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executing_high_water": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_executed": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "pq_item_in_journal": 0 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:56.958 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 4779, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 2104266, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 4779, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 4779, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 2104266, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.959 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_bytes": { 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 38, 2024-03-18T20:32:56.960 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 11466, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 38, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 11466, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-objecter_ops": { 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 1024, 2024-03-18T20:32:56.961 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 38, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 38, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 38, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 38, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.962 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle": { 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:56.963 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 0, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 0, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-write_buf_throttle-0x558dc45d80a0": { 2024-03-18T20:32:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 3758096384, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 2, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 1035, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 2, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 2, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 1035, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:56.965 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:56.966 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:56.966 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.966 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:56.966 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-18T20:32:57.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:56 smithi169 ceph-mon[48529]: pgmap v474: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-18T20:32:57.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:56 smithi098 ceph-mon[43462]: pgmap v474: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-18T20:32:57.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:56 smithi136 ceph-mon[48073]: pgmap v474: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-18T20:32:57.308 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-18T20:32:57.308 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-18T20:32:57.308 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-18T20:32:57.308 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=501 2024-03-18T20:32:57.308 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-18T20:32:57.308 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T20:32:57.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:49bdc51142759622a281d3e5327eebd07cbb8bb3 shell --fsid a6e2b072-e564-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T20:32:57.532 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 145, 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 142, 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 450768, 2024-03-18T20:32:58.585 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 80432, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.025574257, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.008243838, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.013541271, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.003991963, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 141, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.008362326, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000059307 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:58.586 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 450768, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 80432 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 108, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 160, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 95024, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 3, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-18T20:32:58.587 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.006441606, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.004536327, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000628930, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 106, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.005470737, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000051610 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:58.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 95024 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 4, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 4, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 3008, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 448, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.000879581, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000039411, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000841780, 2024-03-18T20:32:58.589 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000824626, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000274875 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3008, 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T20:32:58.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T20:32:58.591 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T20:32:58.592 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 148967, 2024-03-18T20:32:58.593 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 29, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-18T20:32:58.594 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-18T20:32:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 146, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 430295, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-18T20:32:58.596 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 146, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 146, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 430295, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-18T20:32:58.597 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-18T20:32:58.985 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-18T20:32:58.985 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-18T20:32:58.985 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-18T20:32:58.985 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-18T20:32:58.985 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-18T20:32:58.986 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-18T20:32:58.987 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-18T20:32:58.998 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-18T20:32:58.998 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:32:58.998 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:32:59.022 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:32:59.022 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:32:59.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:32:59.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:58 smithi098 ceph-mon[43462]: pgmap v475: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:32:59.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:58 smithi169 ceph-mon[48529]: pgmap v475: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:32:59.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- 172.21.15.98:0/2195549031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4301040b0 msgr2=0x7fc4301064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 --2- 172.21.15.98:0/2195549031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4301040b0 0x7fc4301064a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc420009990 tx=0x7fc420009ca0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- 172.21.15.98:0/2195549031 shutdown_connections 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 --2- 172.21.15.98:0/2195549031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 0x7fc430108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 --2- 172.21.15.98:0/2195549031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc430101780 0x7fc430103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 --2- 172.21.15.98:0/2195549031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4301040b0 0x7fc4301064a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- 172.21.15.98:0/2195549031 >> 172.21.15.98:0/2195549031 conn(0x7fc4300fb0d0 msgr2=0x7fc4300fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- 172.21.15.98:0/2195549031 shutdown_connections 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- 172.21.15.98:0/2195549031 wait complete. 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 Processor -- start 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.222+0000 7fc43644a700 1 -- start start 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4301040b0 0x7fc4300ff730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 0x7fc4300ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430100410 con 0x7fc430101780 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430100580 con 0x7fc4301040b0 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4301a0b00 con 0x7fc4301069e0 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41630/0 (socket says 172.21.15.98:41630) 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 -- 172.21.15.98:0/3266672050 learned_addr learned my addr 172.21.15.98:0/3266672050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc4349e7700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 0x7fc4300ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42f7fe700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4301040b0 0x7fc4300ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 -- 172.21.15.98:0/3266672050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 msgr2=0x7fc4300ffc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 0x7fc4300ffc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 -- 172.21.15.98:0/3266672050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4301040b0 msgr2=0x7fc4300ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4301040b0 0x7fc4300ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 -- 172.21.15.98:0/3266672050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200095f0 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42ffff700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c920 tx=0x7fc42400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424007aa0 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc42400ce70 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4301a0d90 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240053a0 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.223+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4301a11e0 con 0x7fc430101780 2024-03-18T20:32:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.224+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4300fcc50 con 0x7fc430101780 2024-03-18T20:32:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.225+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc424005520 con 0x7fc430101780 2024-03-18T20:32:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.226+0000 7fc42d7fa700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 0x7fc4180a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.226+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7fc4240ccc00 con 0x7fc430101780 2024-03-18T20:32:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.226+0000 7fc42f7fe700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 0x7fc4180a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.226+0000 7fc42f7fe700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 0x7fc4180a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc420003780 tx=0x7fc420005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.230+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc424095fd0 con 0x7fc430101780 2024-03-18T20:32:59.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:58 smithi136 ceph-mon[48073]: pgmap v475: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:32:59.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.435+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc430067a10 con 0x7fc430101780 2024-03-18T20:32:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.436+0000 7fc42d7fa700 1 -- 172.21.15.98:0/3266672050 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24770 (secure 0 0 0) 0x7fc42401d020 con 0x7fc430101780 2024-03-18T20:32:59.441 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:32:59.441 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:26:46.747051+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 msgr2=0x7fc4180a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 0x7fc4180a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc420003780 tx=0x7fc420005fd0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 msgr2=0x7fc4301010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c920 tx=0x7fc42400cc30 comp rx=0 tx=0).stop 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 shutdown_connections 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc4301069e0 0x7fc4300ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc430101780 0x7fc4301010a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7fc4180a1120 0x7fc4180a35e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 --2- 172.21.15.98:0/3266672050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4301040b0 0x7fc4300ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 >> 172.21.15.98:0/3266672050 conn(0x7fc4300fb0d0 msgr2=0x7fc430103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:59.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 shutdown_connections 2024-03-18T20:32:59.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.439+0000 7fc43644a700 1 -- 172.21.15.98:0/3266672050 wait complete. 2024-03-18T20:32:59.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:32:59.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/2311737575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf4100e50 msgr2=0x7f9bf41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/2311737575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf4100e50 0x7f9bf41012b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0099c0 tx=0x7f9bdc009cd0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/2311737575 shutdown_connections 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/2311737575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf41017f0 0x7f9bf41080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/2311737575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf40ffc60 0x7f9bf4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/2311737575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf4100e50 0x7f9bf41012b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/2311737575 >> 172.21.15.98:0/2311737575 conn(0x7f9bf40fb1d0 msgr2=0x7f9bf40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/2311737575 shutdown_connections 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.577+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/2311737575 wait complete. 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 Processor -- start 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 -- start start 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf40ffc60 0x7f9bf4196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf4100e50 0x7f9bf4197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf419e770 con 0x7f9bf41017f0 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf419e8b0 con 0x7f9bf40ffc60 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bfa4bb700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf419ea40 con 0x7f9bf4100e50 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bf8a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bf8a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41640/0 (socket says 172.21.15.98:41640) 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bf8a58700 1 -- 172.21.15.98:0/3456944619 learned_addr learned my addr 172.21.15.98:0/3456944619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:59.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bebfff700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf4100e50 0x7f9bf4197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.578+0000 7f9bf3fff700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf40ffc60 0x7f9bf4196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 -- 172.21.15.98:0/3456944619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf4100e50 msgr2=0x7f9bf4197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf4100e50 0x7f9bf4197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 -- 172.21.15.98:0/3456944619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf40ffc60 msgr2=0x7f9bf4196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf40ffc60 0x7f9bf4196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 -- 172.21.15.98:0/3456944619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc0095f0 con 0x7f9bf41017f0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf8a58700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9bec007ee0 tx=0x7f9bec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00dd20 con 0x7f9bf41017f0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9bec00f040 con 0x7f9bf41017f0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf419ed20 con 0x7f9bf41017f0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec004750 con 0x7f9bf41017f0 2024-03-18T20:32:59.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.579+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf419f1d0 con 0x7f9bf41017f0 2024-03-18T20:32:59.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.580+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf4067af0 con 0x7f9bf41017f0 2024-03-18T20:32:59.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.581+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f9bec003a00 con 0x7f9bf41017f0 2024-03-18T20:32:59.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.582+0000 7f9bf1ffb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 0x7f9be00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.582+0000 7f9bf3fff700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 0x7f9be00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.582+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f9bec0ccc80 con 0x7f9bf41017f0 2024-03-18T20:32:59.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.582+0000 7f9bf3fff700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 0x7f9be00a3660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4100cb0 tx=0x7f9be4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.585+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9bec096050 con 0x7f9bf41017f0 2024-03-18T20:32:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.781+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bf419f3c0 con 0x7f9bf41017f0 2024-03-18T20:32:59.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.781+0000 7f9bf1ffb700 1 -- 172.21.15.98:0/3456944619 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bec023020 con 0x7f9bf41017f0 2024-03-18T20:32:59.786 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 msgr2=0x7f9be00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 0x7f9be00a3660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4100cb0 tx=0x7f9be4009260 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 msgr2=0x7f9bf419e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9bec007ee0 tx=0x7f9bec00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 shutdown_connections 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf41017f0 0x7f9bf419e1a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f9be00a11a0 0x7f9be00a3660 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf40ffc60 0x7f9bf4196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 --2- 172.21.15.98:0/3456944619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf4100e50 0x7f9bf4197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.783+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 >> 172.21.15.98:0/3456944619 conn(0x7f9bf40fb1d0 msgr2=0x7f9bf4067df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.784+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 shutdown_connections 2024-03-18T20:32:59.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.784+0000 7f9bfa4bb700 1 -- 172.21.15.98:0/3456944619 wait complete. 2024-03-18T20:32:59.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 -- 172.21.15.98:0/2078902675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c106a70 msgr2=0x7ffa9c108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/2078902675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c106a70 0x7ffa9c108ee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400b0a0 tx=0x7ffa9400b3b0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 -- 172.21.15.98:0/2078902675 shutdown_connections 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/2078902675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c106a70 0x7ffa9c108ee0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/2078902675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c101810 0x7ffa9c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/2078902675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c104140 0x7ffa9c106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 -- 172.21.15.98:0/2078902675 >> 172.21.15.98:0/2078902675 conn(0x7ffa9c0fb200 msgr2=0x7ffa9c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:32:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 -- 172.21.15.98:0/2078902675 shutdown_connections 2024-03-18T20:32:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.921+0000 7ffaa42f3700 1 -- 172.21.15.98:0/2078902675 wait complete. 2024-03-18T20:32:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 Processor -- start 2024-03-18T20:32:59.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 -- start start 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c101810 0x7ffa9c196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c106a70 0x7ffa9c19e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19e800 con 0x7ffa9c104140 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19e970 con 0x7ffa9c106a70 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa42f3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19eb30 con 0x7ffa9c101810 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa188e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa188e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41648/0 (socket says 172.21.15.98:41648) 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.922+0000 7ffaa188e700 1 -- 172.21.15.98:0/523627179 learned_addr learned my addr 172.21.15.98:0/523627179 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa208f700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c101810 0x7ffa9c196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 -- 172.21.15.98:0/523627179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c101810 msgr2=0x7ffa9c196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa2890700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c106a70 0x7ffa9c19e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c101810 0x7ffa9c196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 -- 172.21.15.98:0/523627179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c106a70 msgr2=0x7ffa9c19e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c106a70 0x7ffa9c19e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 -- 172.21.15.98:0/523627179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94009d00 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa188e700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c920 tx=0x7ffa9000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90007b00 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffa90004cc0 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c19ee10 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90005490 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.923+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c19f290 con 0x7ffa9c104140 2024-03-18T20:32:59.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.924+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c0fcd80 con 0x7ffa9c104140 2024-03-18T20:32:59.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.925+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ffa90004750 con 0x7ffa9c104140 2024-03-18T20:32:59.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.926+0000 7ffa8f7fe700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:32:59.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.926+0000 7ffaa208f700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:32:59.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.926+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7ffa900cccb0 con 0x7ffa9c104140 2024-03-18T20:32:59.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.926+0000 7ffaa208f700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 0x7ffa880a3660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffa98005950 tx=0x7ffa98005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:32:59.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:32:59.929+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa90096020 con 0x7ffa9c104140 2024-03-18T20:33:00.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:32:59 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3266672050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:00.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.127+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa9c02d110 con 0x7ffa9c104140 2024-03-18T20:33:00.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.127+0000 7ffa8f7fe700 1 -- 172.21.15.98:0/523627179 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa9001d0a0 con 0x7ffa9c104140 2024-03-18T20:33:00.134 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 msgr2=0x7ffa880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 0x7ffa880a3660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffa98005950 tx=0x7ffa98005ed0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 msgr2=0x7ffa9c197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c920 tx=0x7ffa9000cce0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 shutdown_connections 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9c106a70 0x7ffa9c19e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa9c101810 0x7ffa9c196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.129+0000 7ffaa42f3700 1 --2- 172.21.15.98:0/523627179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa9c104140 0x7ffa9c197220 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.130+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 >> 172.21.15.98:0/523627179 conn(0x7ffa9c0fb200 msgr2=0x7ffa9c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.130+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 shutdown_connections 2024-03-18T20:33:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.130+0000 7ffaa42f3700 1 -- 172.21.15.98:0/523627179 wait complete. 2024-03-18T20:33:00.136 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:32:59 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3266672050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:00.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:33:00.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:32:59 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3266672050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 -- 172.21.15.98:0/2105389576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 msgr2=0x7f5bb01080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 --2- 172.21.15.98:0/2105389576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb01080a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800b0d0 tx=0x7f5ba800b3e0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 -- 172.21.15.98:0/2105389576 shutdown_connections 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 --2- 172.21.15.98:0/2105389576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb01080a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 --2- 172.21.15.98:0/2105389576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb00ffc60 0x7f5bb0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 --2- 172.21.15.98:0/2105389576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb0100e50 0x7f5bb01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 -- 172.21.15.98:0/2105389576 >> 172.21.15.98:0/2105389576 conn(0x7f5bb00fb1d0 msgr2=0x7f5bb00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 -- 172.21.15.98:0/2105389576 shutdown_connections 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.268+0000 7f5bb5045700 1 -- 172.21.15.98:0/2105389576 wait complete. 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 Processor -- start 2024-03-18T20:33:00.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 -- start start 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb00ffc60 0x7f5bb0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0100e50 0x7f5bb019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb019e850 con 0x7f5bb01017f0 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb019e9c0 con 0x7f5bb0100e50 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bb5045700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb019eb80 con 0x7f5bb00ffc60 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5baf59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5baf59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41664/0 (socket says 172.21.15.98:41664) 2024-03-18T20:33:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5baf59e700 1 -- 172.21.15.98:0/1353574282 learned_addr learned my addr 172.21.15.98:0/1353574282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5bae59c700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0100e50 0x7f5bb019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.269+0000 7f5baed9d700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb00ffc60 0x7f5bb0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 -- 172.21.15.98:0/1353574282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb00ffc60 msgr2=0x7f5bb0068930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb00ffc60 0x7f5bb0068930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 -- 172.21.15.98:0/1353574282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0100e50 msgr2=0x7f5bb019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0100e50 0x7f5bb019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 -- 172.21.15.98:0/1353574282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8009d00 con 0x7f5bb01017f0 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5baf59e700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8007b60 tx=0x7f5ba80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba800e040 con 0x7f5bb01017f0 2024-03-18T20:33:00.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb019eda0 con 0x7f5bb01017f0 2024-03-18T20:33:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ba801dd20 con 0x7f5bb01017f0 2024-03-18T20:33:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8012dc0 con 0x7f5bb01017f0 2024-03-18T20:33:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.270+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb019f210 con 0x7f5bb01017f0 2024-03-18T20:33:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.271+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb01070e0 con 0x7f5bb01017f0 2024-03-18T20:33:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.272+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f5ba801d890 con 0x7f5bb01017f0 2024-03-18T20:33:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.273+0000 7f5b97fff700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 0x7f5b980a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.273+0000 7f5baed9d700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 0x7f5b980a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.273+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f5ba80cedf0 con 0x7f5bb01017f0 2024-03-18T20:33:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.273+0000 7f5baed9d700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 0x7f5b980a3660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0100cb0 tx=0x7f5ba400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.276+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ba80981f0 con 0x7f5bb01017f0 2024-03-18T20:33:00.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.471+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5bb002d0f0 con 0x7f5bb01017f0 2024-03-18T20:33:00.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.471+0000 7f5b97fff700 1 -- 172.21.15.98:0/1353574282 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ba809b810 con 0x7f5bb01017f0 2024-03-18T20:33:00.476 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 msgr2=0x7f5b980a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 0x7f5b980a3660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0100cb0 tx=0x7f5ba400b410 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 msgr2=0x7f5bb019e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8007b60 tx=0x7f5ba80094c0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 shutdown_connections 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bb01017f0 0x7f5bb019e060 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5bb00ffc60 0x7f5bb0068930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f5b980a11a0 0x7f5b980a3660 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 --2- 172.21.15.98:0/1353574282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0100e50 0x7f5bb019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.473+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 >> 172.21.15.98:0/1353574282 conn(0x7f5bb00fb1d0 msgr2=0x7f5bb0104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.474+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 shutdown_connections 2024-03-18T20:33:00.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.474+0000 7f5bb5045700 1 -- 172.21.15.98:0/1353574282 wait complete. 2024-03-18T20:33:00.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:33:00.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 -- 172.21.15.98:0/3295240877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4434100860 msgr2=0x7f4434108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 --2- 172.21.15.98:0/3295240877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4434100860 0x7f4434108100 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f442c00b0a0 tx=0x7f442c00b3b0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 -- 172.21.15.98:0/3295240877 shutdown_connections 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 --2- 172.21.15.98:0/3295240877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4434100860 0x7f4434108100 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 --2- 172.21.15.98:0/3295240877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44341012e0 0x7f4434101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 --2- 172.21.15.98:0/3295240877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44340ffea0 0x7f4434100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 -- 172.21.15.98:0/3295240877 >> 172.21.15.98:0/3295240877 conn(0x7f44340fa950 msgr2=0x7f44340fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 -- 172.21.15.98:0/3295240877 shutdown_connections 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.610+0000 7f443a621700 1 -- 172.21.15.98:0/3295240877 wait complete. 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 Processor -- start 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 -- start start 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 0x7f443419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44341012e0 0x7f443419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443419e810 con 0x7f44340ffea0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443419e980 con 0x7f4434100860 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f443a621700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443419eb40 con 0x7f44341012e0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f4433fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f4433fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41682/0 (socket says 172.21.15.98:41682) 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.611+0000 7f4433fff700 1 -- 172.21.15.98:0/2811083790 learned_addr learned my addr 172.21.15.98:0/2811083790 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f44337fe700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 0x7f443419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 -- 172.21.15.98:0/2811083790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44341012e0 msgr2=0x7f443419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4438bbe700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44341012e0 0x7f443419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44341012e0 0x7f443419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 -- 172.21.15.98:0/2811083790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 msgr2=0x7f443419ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 0x7f443419ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 -- 172.21.15.98:0/2811083790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c009d00 con 0x7f44340ffea0 2024-03-18T20:33:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f44337fe700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 0x7f443419ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f4433fff700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f442400e9a0 tx=0x7f442400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424009790 con 0x7f44340ffea0 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443419edc0 con 0x7f44340ffea0 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f442400cbd0 con 0x7f44340ffea0 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424018680 con 0x7f44340ffea0 2024-03-18T20:33:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.612+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443419f210 con 0x7f44340ffea0 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.613+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4434107180 con 0x7f44340ffea0 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.614+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f442400cd40 con 0x7f44340ffea0 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.615+0000 7f44317fa700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 0x7f441c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.615+0000 7f44337fe700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 0x7f441c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.615+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f4424014070 con 0x7f44340ffea0 2024-03-18T20:33:00.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.615+0000 7f44337fe700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 0x7f441c0a35e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4428005fd0 tx=0x7f4428005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.618+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4424096580 con 0x7f44340ffea0 2024-03-18T20:33:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.816+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f443402cdc0 con 0x7f44340ffea0 2024-03-18T20:33:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.816+0000 7f44317fa700 1 -- 172.21.15.98:0/2811083790 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4424026020 con 0x7f44340ffea0 2024-03-18T20:33:00.821 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:33:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 msgr2=0x7f441c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 0x7f441c0a35e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4428005fd0 tx=0x7f4428005ee0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 msgr2=0x7f44340ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f442400e9a0 tx=0x7f442400ed60 comp rx=0 tx=0).stop 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 shutdown_connections 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44341012e0 0x7f443419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44340ffea0 0x7f44340ff8d0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f441c0a1120 0x7f441c0a35e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 --2- 172.21.15.98:0/2811083790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4434100860 0x7f443419ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 >> 172.21.15.98:0/2811083790 conn(0x7f44340fa950 msgr2=0x7f44341042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 shutdown_connections 2024-03-18T20:33:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.819+0000 7f443a621700 1 -- 172.21.15.98:0/2811083790 wait complete. 2024-03-18T20:33:00.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:33:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/574339111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 msgr2=0x7f3fd8108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/574339111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8108120 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000b0a0 tx=0x7f3fd000b3b0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/574339111 shutdown_connections 2024-03-18T20:33:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/574339111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8108120 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/574339111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd8101270 0x7f3fd8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/574339111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/574339111 >> 172.21.15.98:0/574339111 conn(0x7f3fd80fa950 msgr2=0x7f3fd80fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/574339111 shutdown_connections 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.955+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/574339111 wait complete. 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 Processor -- start 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 -- start start 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd8101270 0x7f3fd819d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd819d7c0 con 0x7f3fd8100880 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd819d930 con 0x7f3fd8101270 2024-03-18T20:33:00.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.956+0000 7f3fdf6b1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd819daf0 con 0x7f3fd80ffec0 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41702/0 (socket says 172.21.15.98:41702) 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 -- 172.21.15.98:0/1465193994 learned_addr learned my addr 172.21.15.98:0/1465193994 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fddc4e700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd8101270 0x7f3fd819d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdd44d700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 -- 172.21.15.98:0/1465193994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 msgr2=0x7f3fd8195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 -- 172.21.15.98:0/1465193994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd8101270 msgr2=0x7f3fd819d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd8101270 0x7f3fd819d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 -- 172.21.15.98:0/1465193994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0009d00 con 0x7f3fd8100880 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdd44d700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdcc4c700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400d980 tx=0x7f3fd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4004a30 con 0x7f3fd8100880 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3fd40054b0 con 0x7f3fd8100880 2024-03-18T20:33:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd819ddd0 con 0x7f3fd8100880 2024-03-18T20:33:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4009bf0 con 0x7f3fd8100880 2024-03-18T20:33:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.957+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd819e1d0 con 0x7f3fd8100880 2024-03-18T20:33:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.958+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd8107160 con 0x7f3fd8100880 2024-03-18T20:33:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.959+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f3fd4009d50 con 0x7f3fd8100880 2024-03-18T20:33:00.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.960+0000 7f3fca7fc700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 0x7f3fc40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:00.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.960+0000 7f3fdd44d700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 0x7f3fc40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:00.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.960+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f3fd40ccf90 con 0x7f3fd8100880 2024-03-18T20:33:00.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.960+0000 7f3fdd44d700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 0x7f3fc40a35e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8197150 tx=0x7f3fcc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:00.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:00.963+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fd4096300 con 0x7f3fd8100880 2024-03-18T20:33:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:00 smithi098 ceph-mon[43462]: pgmap v476: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-18T20:33:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3456944619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/523627179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:00 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1353574282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:00 smithi169 ceph-mon[48529]: pgmap v476: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-18T20:33:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3456944619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/523627179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:00 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1353574282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.187+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3fd802d0f0 con 0x7f3fd8100880 2024-03-18T20:33:01.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.187+0000 7f3fca7fc700 1 -- 172.21.15.98:0/1465193994 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7f3fd4017070 con 0x7f3fd8100880 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2681086384 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3671856112 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1417871753 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2768304991 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2082705134 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:01.192 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3238391768 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 msgr2=0x7f3fc40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 0x7f3fc40a35e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8197150 tx=0x7f3fcc008040 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 msgr2=0x7f3fd8196210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400d980 tx=0x7f3fd400dc90 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 shutdown_connections 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f3fc40a1120 0x7f3fc40a35e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fd8101270 0x7f3fd819d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fd80ffec0 0x7f3fd8195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 --2- 172.21.15.98:0/1465193994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8100880 0x7f3fd8196210 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 >> 172.21.15.98:0/1465193994 conn(0x7f3fd80fa950 msgr2=0x7f3fd81042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:01.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.189+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 shutdown_connections 2024-03-18T20:33:01.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.190+0000 7f3fdf6b1700 1 -- 172.21.15.98:0/1465193994 wait complete. 2024-03-18T20:33:01.195 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-18T20:33:01.207 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-18T20:33:01.207 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:33:01.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:33:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:00 smithi136 ceph-mon[48073]: pgmap v476: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-18T20:33:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3456944619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/523627179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:00 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1353574282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 -- 172.21.15.98:0/216554279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5730100d60 msgr2=0x7f57301011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 --2- 172.21.15.98:0/216554279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5730100d60 0x7f57301011c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f57180099c0 tx=0x7f5718009cd0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 -- 172.21.15.98:0/216554279 shutdown_connections 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 --2- 172.21.15.98:0/216554279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f5730107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 --2- 172.21.15.98:0/216554279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57300ffb70 0x7f57300fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 --2- 172.21.15.98:0/216554279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5730100d60 0x7f57301011c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 -- 172.21.15.98:0/216554279 >> 172.21.15.98:0/216554279 conn(0x7f57300fb100 msgr2=0x7f57300fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 -- 172.21.15.98:0/216554279 shutdown_connections 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.384+0000 7f5734f83700 1 -- 172.21.15.98:0/216554279 wait complete. 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 Processor -- start 2024-03-18T20:33:01.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 -- start start 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 0x7f5730195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f573019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573019d530 con 0x7f57300ffb70 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573019d670 con 0x7f5730101700 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f5734f83700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573019d800 con 0x7f5730100d60 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41714/0 (socket says 172.21.15.98:41714) 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 -- 172.21.15.98:0/4152457983 learned_addr learned my addr 172.21.15.98:0/4152457983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:01.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572ed9d700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f573019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572dd9b700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 0x7f5730195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 -- 172.21.15.98:0/4152457983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 msgr2=0x7f5730195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 0x7f5730195f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 -- 172.21.15.98:0/4152457983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 msgr2=0x7f573019cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f573019cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572e59c700 1 -- 172.21.15.98:0/4152457983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180095f0 con 0x7f57300ffb70 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572ed9d700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f573019cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.385+0000 7f572dd9b700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 0x7f5730195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f572e59c700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f572000d980 tx=0x7f572000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572000de60 con 0x7f57300ffb70 2024-03-18T20:33:01.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f57200044b0 con 0x7f57300ffb70 2024-03-18T20:33:01.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573019dae0 con 0x7f57300ffb70 2024-03-18T20:33:01.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720003b40 con 0x7f57300ffb70 2024-03-18T20:33:01.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.386+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5730069230 con 0x7f57300ffb70 2024-03-18T20:33:01.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.387+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5730067180 con 0x7f57300ffb70 2024-03-18T20:33:01.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.388+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f5720004870 con 0x7f57300ffb70 2024-03-18T20:33:01.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.389+0000 7f57277fe700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 0x7f571c0a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.389+0000 7f572dd9b700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 0x7f571c0a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.389+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f57200ccdb0 con 0x7f57300ffb70 2024-03-18T20:33:01.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.389+0000 7f572dd9b700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 0x7f571c0a57f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5718006010 tx=0x7f5718009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:01.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.391+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5720096120 con 0x7f57300ffb70 2024-03-18T20:33:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.601+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5730069c80 con 0x7f57300ffb70 2024-03-18T20:33:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.603+0000 7f57277fe700 1 -- 172.21.15.98:0/4152457983 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24770 (secure 0 0 0) 0x7f5720009940 con 0x7f57300ffb70 2024-03-18T20:33:01.607 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-18T20:33:01.608 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"a6e2b072-e564-11ee-95c9-87774f69a715","created":"2024-03-18T20:18:57.763866+0000","modified":"2024-03-18T20:26:46.747051+0000","last_up_change":"2024-03-18T20:24:08.629635+0000","last_in_change":"2024-03-18T20:23:52.584860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T20:21:33.821476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:24:39.571269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T20:24:40.507865+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T20:24:43.651036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d91034c-3502-4016-9090-515df19fa756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6803","nonce":363521185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6805","nonce":363521185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6809","nonce":363521185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":363521185},{"type":"v1","addr":"172.21.15.98:6807","nonce":363521185}]},"public_addr":"172.21.15.98:6803/363521185","cluster_addr":"172.21.15.98:6805/363521185","heartbeat_back_addr":"172.21.15.98:6809/363521185","heartbeat_front_addr":"172.21.15.98:6807/363521185","state":["exists","up"]},{"osd":1,"uuid":"cf985f1e-4bad-4e71-9cf6-abea91ae0378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6801","nonce":1522927033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6803","nonce":1522927033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6807","nonce":1522927033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1522927033},{"type":"v1","addr":"172.21.15.136:6805","nonce":1522927033}]},"public_addr":"172.21.15.136:6801/1522927033","cluster_addr":"172.21.15.136:6803/1522927033","heartbeat_back_addr":"172.21.15.136:6807/1522927033","heartbeat_front_addr":"172.21.15.136:6805/1522927033","state":["exists","up"]},{"osd":2,"uuid":"14a75dc5-6809-4c9f-9222-8e6a71c62a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6801","nonce":151861743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6803","nonce":151861743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6807","nonce":151861743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":151861743},{"type":"v1","addr":"172.21.15.169:6805","nonce":151861743}]},"public_addr":"172.21.15.169:6801/151861743","cluster_addr":"172.21.15.169:6803/151861743","heartbeat_back_addr":"172.21.15.169:6807/151861743","heartbeat_front_addr":"172.21.15.169:6805/151861743","state":["exists","up"]},{"osd":3,"uuid":"dc4d0794-a030-42e7-8cdd-6e2c636e56ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6811","nonce":2727695950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6813","nonce":2727695950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6817","nonce":2727695950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2727695950},{"type":"v1","addr":"172.21.15.98:6815","nonce":2727695950}]},"public_addr":"172.21.15.98:6811/2727695950","cluster_addr":"172.21.15.98:6813/2727695950","heartbeat_back_addr":"172.21.15.98:6817/2727695950","heartbeat_front_addr":"172.21.15.98:6815/2727695950","state":["exists","up"]},{"osd":4,"uuid":"63fde25b-a451-4031-8795-9a05f0f2a29d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6809","nonce":4061852500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6811","nonce":4061852500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6815","nonce":4061852500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":4061852500},{"type":"v1","addr":"172.21.15.136:6813","nonce":4061852500}]},"public_addr":"172.21.15.136:6809/4061852500","cluster_addr":"172.21.15.136:6811/4061852500","heartbeat_back_addr":"172.21.15.136:6815/4061852500","heartbeat_front_addr":"172.21.15.136:6813/4061852500","state":["exists","up"]},{"osd":5,"uuid":"62b9ea73-b07c-4aa1-8d5d-57fdf0204185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6809","nonce":123568177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6811","nonce":123568177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6815","nonce":123568177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":123568177},{"type":"v1","addr":"172.21.15.169:6813","nonce":123568177}]},"public_addr":"172.21.15.169:6809/123568177","cluster_addr":"172.21.15.169:6811/123568177","heartbeat_back_addr":"172.21.15.169:6815/123568177","heartbeat_front_addr":"172.21.15.169:6813/123568177","state":["exists","up"]},{"osd":6,"uuid":"2aaa088e-a86d-43f6-bbf6-7fdbf0b1cef1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6819","nonce":2163470647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6821","nonce":2163470647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6825","nonce":2163470647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2163470647},{"type":"v1","addr":"172.21.15.98:6823","nonce":2163470647}]},"public_addr":"172.21.15.98:6819/2163470647","cluster_addr":"172.21.15.98:6821/2163470647","heartbeat_back_addr":"172.21.15.98:6825/2163470647","heartbeat_front_addr":"172.21.15.98:6823/2163470647","state":["exists","up"]},{"osd":7,"uuid":"df65208c-eee4-455f-b0e0-9a42f261531e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6817","nonce":1282488647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6819","nonce":1282488647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6823","nonce":1282488647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1282488647},{"type":"v1","addr":"172.21.15.136:6821","nonce":1282488647}]},"public_addr":"172.21.15.136:6817/1282488647","cluster_addr":"172.21.15.136:6819/1282488647","heartbeat_back_addr":"172.21.15.136:6823/1282488647","heartbeat_front_addr":"172.21.15.136:6821/1282488647","state":["exists","up"]},{"osd":8,"uuid":"5ff34b6f-b1af-4751-afda-9fa4bc03e881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6817","nonce":3298544672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6819","nonce":3298544672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6823","nonce":3298544672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3298544672},{"type":"v1","addr":"172.21.15.169:6821","nonce":3298544672}]},"public_addr":"172.21.15.169:6817/3298544672","cluster_addr":"172.21.15.169:6819/3298544672","heartbeat_back_addr":"172.21.15.169:6823/3298544672","heartbeat_front_addr":"172.21.15.169:6821/3298544672","state":["exists","up"]},{"osd":9,"uuid":"5969a1df-7157-4720-bb4e-d499f9a2d3d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6827","nonce":1957300736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6829","nonce":1957300736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6833","nonce":1957300736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1957300736},{"type":"v1","addr":"172.21.15.98:6831","nonce":1957300736}]},"public_addr":"172.21.15.98:6827/1957300736","cluster_addr":"172.21.15.98:6829/1957300736","heartbeat_back_addr":"172.21.15.98:6833/1957300736","heartbeat_front_addr":"172.21.15.98:6831/1957300736","state":["exists","up"]},{"osd":10,"uuid":"dbb5b70e-d702-48c4-9811-d1d75e09de82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6825","nonce":3492008064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6827","nonce":3492008064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6831","nonce":3492008064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3492008064},{"type":"v1","addr":"172.21.15.136:6829","nonce":3492008064}]},"public_addr":"172.21.15.136:6825/3492008064","cluster_addr":"172.21.15.136:6827/3492008064","heartbeat_back_addr":"172.21.15.136:6831/3492008064","heartbeat_front_addr":"172.21.15.136:6829/3492008064","state":["exists","up"]},{"osd":11,"uuid":"50289373-3457-45bd-8c72-de8ae15b6ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6825","nonce":2513253907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6827","nonce":2513253907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6831","nonce":2513253907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":2513253907},{"type":"v1","addr":"172.21.15.169:6829","nonce":2513253907}]},"public_addr":"172.21.15.169:6825/2513253907","cluster_addr":"172.21.15.169:6827/2513253907","heartbeat_back_addr":"172.21.15.169:6831/2513253907","heartbeat_front_addr":"172.21.15.169:6829/2513253907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:20:55.527697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:13.059498+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:30.628282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:21:47.752278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:04.908248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:22.685692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:39.914959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:22:57.727191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:15.150884+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:32.497178+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:23:49.738103+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T20:24:07.307330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3b","mappings":[{"from":3,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/1287611061":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/2681086384":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/3671856112":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/1417871753":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2768304991":"2024-03-19T20:19:45.512552+0000","172.21.15.98:6800/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:6801/479479747":"2024-03-19T20:19:25.585401+0000","172.21.15.98:0/2082705134":"2024-03-19T20:19:45.512552+0000","172.21.15.98:0/3238391768":"2024-03-19T20:19:25.585401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:33:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 msgr2=0x7f571c0a57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 0x7f571c0a57f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5718006010 tx=0x7f5718009510 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 msgr2=0x7f5730195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f572000d980 tx=0x7f572000dd40 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 shutdown_connections 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f571c0a3330 0x7f571c0a57f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5730101700 0x7f573019cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57300ffb70 0x7f5730195a00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 --2- 172.21.15.98:0/4152457983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5730100d60 0x7f5730195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 >> 172.21.15.98:0/4152457983 conn(0x7f57300fb100 msgr2=0x7f57301041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 shutdown_connections 2024-03-18T20:33:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.605+0000 7f5734f83700 1 -- 172.21.15.98:0/4152457983 wait complete. 2024-03-18T20:33:01.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T20:21:33.821476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:24:39.571269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:24:40.507865+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T20:24:43.651036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T20:33:01.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:33:01.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 -- 172.21.15.98:0/1570617537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 msgr2=0x7f21601011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 --2- 172.21.15.98:0/1570617537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f21601011a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f215c0099c0 tx=0x7f215c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 -- 172.21.15.98:0/1570617537 shutdown_connections 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 --2- 172.21.15.98:0/1570617537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2160102910 0x7f21601091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 --2- 172.21.15.98:0/1570617537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f21601011a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 --2- 172.21.15.98:0/1570617537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160101f70 0x7f21601023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 -- 172.21.15.98:0/1570617537 >> 172.21.15.98:0/1570617537 conn(0x7f21600fc2f0 msgr2=0x7f21600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 -- 172.21.15.98:0/1570617537 shutdown_connections 2024-03-18T20:33:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.743+0000 7f216789a700 1 -- 172.21.15.98:0/1570617537 wait complete. 2024-03-18T20:33:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 Processor -- start 2024-03-18T20:33:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 -- start start 2024-03-18T20:33:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2160101f70 0x7f21600731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160102910 0x7f21600736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2160073e80 con 0x7f2160100d80 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2160073ff0 con 0x7f2160101f70 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f216789a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21601a0b60 con 0x7f2160102910 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41736/0 (socket says 172.21.15.98:41736) 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 -- 172.21.15.98:0/1623443294 learned_addr learned my addr 172.21.15.98:0/1623443294 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165e37700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160102910 0x7f21600736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 -- 172.21.15.98:0/1623443294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160102910 msgr2=0x7f21600736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160102910 0x7f21600736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 -- 172.21.15.98:0/1623443294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2160101f70 msgr2=0x7f21600731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2160101f70 0x7f21600731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 -- 172.21.15.98:0/1623443294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0095f0 con 0x7f2160100d80 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.744+0000 7f2165636700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f215c0098e0 tx=0x7f215c003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c01d070 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f215c00bd50 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21601a0d90 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c022d90 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21601a1190 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.745+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21601081b0 con 0x7f2160100d80 2024-03-18T20:33:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.747+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f215c022470 con 0x7f2160100d80 2024-03-18T20:33:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.747+0000 7f21567fc700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 0x7f214c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.747+0000 7f2164e35700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 0x7f214c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.747+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f215c0ce9d0 con 0x7f2160100d80 2024-03-18T20:33:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.748+0000 7f2164e35700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 0x7f214c0a35e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2150005fd0 tx=0x7f2150005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.751+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f215c097e30 con 0x7f2160100d80 2024-03-18T20:33:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.948+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f216002cdc0 con 0x7f2160100d80 2024-03-18T20:33:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.949+0000 7f21567fc700 1 -- 172.21.15.98:0/1623443294 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f215c027020 con 0x7f2160100d80 2024-03-18T20:33:01.953 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 msgr2=0x7f214c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 0x7f214c0a35e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2150005fd0 tx=0x7f2150005d00 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 msgr2=0x7f2160074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f215c0098e0 tx=0x7f215c003b80 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 shutdown_connections 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2160102910 0x7f21600736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2160100d80 0x7f2160074b10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f214c0a1120 0x7f214c0a35e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 --2- 172.21.15.98:0/1623443294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2160101f70 0x7f21600731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 >> 172.21.15.98:0/1623443294 conn(0x7f21600fc2f0 msgr2=0x7f21601052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 shutdown_connections 2024-03-18T20:33:01.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:01.951+0000 7f216789a700 1 -- 172.21.15.98:0/1623443294 wait complete. 2024-03-18T20:33:01.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/2811083790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:01.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1465193994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:01.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:01 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/4152457983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:01.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:33:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 -- 172.21.15.98:0/301577952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8101ff0 msgr2=0x7f98a8102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 --2- 172.21.15.98:0/301577952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8101ff0 0x7f98a8102450 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9898009990 tx=0x7f9898009ca0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 -- 172.21.15.98:0/301577952 shutdown_connections 2024-03-18T20:33:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 --2- 172.21.15.98:0/301577952 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a8102990 0x7f98a8109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 --2- 172.21.15.98:0/301577952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a80ff020 0x7f98a80ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 --2- 172.21.15.98:0/301577952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8101ff0 0x7f98a8102450 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 -- 172.21.15.98:0/301577952 >> 172.21.15.98:0/301577952 conn(0x7f98a80fa950 msgr2=0x7f98a80fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 -- 172.21.15.98:0/301577952 shutdown_connections 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.089+0000 7f98acb23700 1 -- 172.21.15.98:0/301577952 wait complete. 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 Processor -- start 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 -- start start 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a80ff020 0x7f98a8196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 0x7f98a8197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a819e720 con 0x7f98a8102990 2024-03-18T20:33:02.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a819e860 con 0x7f98a8101ff0 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98acb23700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a819e9f0 con 0x7f98a80ff020 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41742/0 (socket says 172.21.15.98:41742) 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98a6d9d700 1 -- 172.21.15.98:0/116493124 learned_addr learned my addr 172.21.15.98:0/116493124 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.090+0000 7f98a659c700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a80ff020 0x7f98a8196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 -- 172.21.15.98:0/116493124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a80ff020 msgr2=0x7f98a8196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a5d9b700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 0x7f98a8197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a80ff020 0x7f98a8196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 -- 172.21.15.98:0/116493124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 msgr2=0x7f98a8197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 0x7f98a8197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 -- 172.21.15.98:0/116493124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980095f0 con 0x7f98a8102990 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a5d9b700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 0x7f98a8197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:33:02.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98a6d9d700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f98a0007ee0 tx=0x7f98a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0017070 con 0x7f98a8102990 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98a000f040 con 0x7f98a8102990 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a819ecd0 con 0x7f98a8102990 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0005710 con 0x7f98a8102990 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.091+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a819f190 con 0x7f98a8102990 2024-03-18T20:33:02.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.092+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a8108320 con 0x7f98a8102990 2024-03-18T20:33:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.093+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f98a0005870 con 0x7f98a8102990 2024-03-18T20:33:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.093+0000 7f989f7fe700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 0x7f98940a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.094+0000 7f98a659c700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 0x7f98940a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.094+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f98a00cc3f0 con 0x7f98a8102990 2024-03-18T20:33:02.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.094+0000 7f98a659c700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 0x7f98940a3660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f98a8067bf0 tx=0x7f9890007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.098+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98a00957c0 con 0x7f98a8102990 2024-03-18T20:33:02.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/2811083790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:02.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1465193994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:02.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:01 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/4152457983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:02.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/2811083790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:02.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1465193994' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:02.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:01 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/4152457983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T20:33:02.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.290+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98a802ce70 con 0x7f98a8102990 2024-03-18T20:33:02.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.290+0000 7f989f7fe700 1 -- 172.21.15.98:0/116493124 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98a0098de0 con 0x7f98a8102990 2024-03-18T20:33:02.295 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 msgr2=0x7f98940a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 0x7f98940a3660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f98a8067bf0 tx=0x7f9890007400 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 msgr2=0x7f98a819e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f98a0007ee0 tx=0x7f98a000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 shutdown_connections 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a8102990 0x7f98a819e150 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a80ff020 0x7f98a8196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f98940a11a0 0x7f98940a3660 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 --2- 172.21.15.98:0/116493124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a8101ff0 0x7f98a8197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.292+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 >> 172.21.15.98:0/116493124 conn(0x7f98a80fa950 msgr2=0x7f98a8105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.293+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 shutdown_connections 2024-03-18T20:33:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.293+0000 7f98acb23700 1 -- 172.21.15.98:0/116493124 wait complete. 2024-03-18T20:33:02.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 -- 172.21.15.98:0/758826115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff484100f70 msgr2=0x7ff484107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 --2- 172.21.15.98:0/758826115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff484100f70 0x7ff484107820 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00b0d0 tx=0x7ff47c00b3e0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 -- 172.21.15.98:0/758826115 shutdown_connections 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 --2- 172.21.15.98:0/758826115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff484100f70 0x7ff484107820 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 --2- 172.21.15.98:0/758826115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 --2- 172.21.15.98:0/758826115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4841005d0 0x7ff484100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 -- 172.21.15.98:0/758826115 >> 172.21.15.98:0/758826115 conn(0x7ff4840fa950 msgr2=0x7ff4840fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.430+0000 7ff48afbf700 1 -- 172.21.15.98:0/758826115 shutdown_connections 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 -- 172.21.15.98:0/758826115 wait complete. 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 Processor -- start 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 -- start start 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff484100f70 0x7ff48419e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419e820 con 0x7ff4841005d0 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419e990 con 0x7ff484100f70 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48afbf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419eb50 con 0x7ff4840ff3e0 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff483fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff483fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41768/0 (socket says 172.21.15.98:41768) 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff483fff700 1 -- 172.21.15.98:0/254436949 learned_addr learned my addr 172.21.15.98:0/254436949 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.431+0000 7ff48955c700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff484100f70 0x7ff48419e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff488d5b700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 -- 172.21.15.98:0/254436949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 msgr2=0x7ff4840fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 -- 172.21.15.98:0/254436949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff484100f70 msgr2=0x7ff48419e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff484100f70 0x7ff48419e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 -- 172.21.15.98:0/254436949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c009d00 con 0x7ff4841005d0 2024-03-18T20:33:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff488d5b700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff483fff700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9a0 tx=0x7ff47000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470009790 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48419ee30 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff47000cbd0 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470018680 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.432+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48419f2b0 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.433+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff484107090 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.434+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ff47000cd40 con 0x7ff4841005d0 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.435+0000 7ff481ffb700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 0x7ff4740a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.435+0000 7ff488d5b700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 0x7ff4740a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.435+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7ff470014070 con 0x7ff4841005d0 2024-03-18T20:33:02.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.435+0000 7ff488d5b700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 0x7ff4740a3660 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff484100430 tx=0x7ff47800b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.439+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff4700529f0 con 0x7ff4841005d0 2024-03-18T20:33:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.637+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4840619c0 con 0x7ff4841005d0 2024-03-18T20:33:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.637+0000 7ff481ffb700 1 -- 172.21.15.98:0/254436949 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff470096270 con 0x7ff4841005d0 2024-03-18T20:33:02.641 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.639+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 msgr2=0x7ff4740a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.639+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 0x7ff4740a3660 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff484100430 tx=0x7ff47800b670 comp rx=0 tx=0).stop 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.639+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 msgr2=0x7ff48419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.639+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9a0 tx=0x7ff47000ecb0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 shutdown_connections 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7ff4740a11a0 0x7ff4740a3660 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff484100f70 0x7ff48419e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4840ff3e0 0x7ff4840fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 --2- 172.21.15.98:0/254436949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4841005d0 0x7ff48419ba30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 >> 172.21.15.98:0/254436949 conn(0x7ff4840fa950 msgr2=0x7ff484101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 shutdown_connections 2024-03-18T20:33:02.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.640+0000 7ff48afbf700 1 -- 172.21.15.98:0/254436949 wait complete. 2024-03-18T20:33:02.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 -- 172.21.15.98:0/1399446862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048100e50 msgr2=0x7f70481012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 --2- 172.21.15.98:0/1399446862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048100e50 0x7f70481012b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f703c0099c0 tx=0x7f703c009cd0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 -- 172.21.15.98:0/1399446862 shutdown_connections 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 --2- 172.21.15.98:0/1399446862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70481017f0 0x7f70481080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 --2- 172.21.15.98:0/1399446862 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70480ffc60 0x7f7048100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 --2- 172.21.15.98:0/1399446862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7048100e50 0x7f70481012b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 -- 172.21.15.98:0/1399446862 >> 172.21.15.98:0/1399446862 conn(0x7f70480fb1d0 msgr2=0x7f70480fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 -- 172.21.15.98:0/1399446862 shutdown_connections 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 -- 172.21.15.98:0/1399446862 wait complete. 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.779+0000 7f704de55700 1 Processor -- start 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 -- start start 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70480ffc60 0x7f7048196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7048100e50 0x7f70481971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704819e770 con 0x7f70481017f0 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704819e8e0 con 0x7f70480ffc60 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f704de55700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704819eaa0 con 0x7f7048100e50 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41780/0 (socket says 172.21.15.98:41780) 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 -- 172.21.15.98:0/3269986826 learned_addr learned my addr 172.21.15.98:0/3269986826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7046ffd700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7048100e50 0x7f70481971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f70477fe700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70480ffc60 0x7f7048196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 -- 172.21.15.98:0/3269986826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7048100e50 msgr2=0x7f70481971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7048100e50 0x7f70481971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 -- 172.21.15.98:0/3269986826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70480ffc60 msgr2=0x7f7048196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70480ffc60 0x7f7048196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 -- 172.21.15.98:0/3269986826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0095f0 con 0x7f70481017f0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7047fff700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f704000ef30 tx=0x7f704000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040018070 con 0x7f70481017f0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.780+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f704000f040 con 0x7f70481017f0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.781+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704819ed20 con 0x7f70481017f0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.781+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000cbc0 con 0x7f70481017f0 2024-03-18T20:33:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.781+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704819f180 con 0x7f70481017f0 2024-03-18T20:33:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.781+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7048067af0 con 0x7f70481017f0 2024-03-18T20:33:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.782+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f704000cd80 con 0x7f70481017f0 2024-03-18T20:33:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.783+0000 7f7044ff9700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 0x7f70300a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.783+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f70400cc730 con 0x7f70481017f0 2024-03-18T20:33:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.783+0000 7f70477fe700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 0x7f70300a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.783+0000 7f70477fe700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 0x7f70300a35e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7048198100 tx=0x7f7038007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.787+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7040095b90 con 0x7f70481017f0 2024-03-18T20:33:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.982+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f704819f310 con 0x7f70481017f0 2024-03-18T20:33:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.982+0000 7f7044ff9700 1 -- 172.21.15.98:0/3269986826 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f70400991b0 con 0x7f70481017f0 2024-03-18T20:33:02.987 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 msgr2=0x7f70300a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 0x7f70300a35e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7048198100 tx=0x7f7038007400 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 msgr2=0x7f704819e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f704000ef30 tx=0x7f704000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 shutdown_connections 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70481017f0 0x7f704819e1a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f70300a1120 0x7f70300a35e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70480ffc60 0x7f7048196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 --2- 172.21.15.98:0/3269986826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7048100e50 0x7f70481971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.984+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 >> 172.21.15.98:0/3269986826 conn(0x7f70480fb1d0 msgr2=0x7f7048067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.985+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 shutdown_connections 2024-03-18T20:33:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:02.985+0000 7f704de55700 1 -- 172.21.15.98:0/3269986826 wait complete. 2024-03-18T20:33:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:02 smithi098 ceph-mon[43462]: pgmap v477: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:33:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:02 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/1623443294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:02 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/116493124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:02 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/254436949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- 172.21.15.98:0/3449927471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c100e80 msgr2=0x7f079c1012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- 172.21.15.98:0/3449927471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c100e80 0x7f079c1012e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f07980099c0 tx=0x7f0798009cd0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- 172.21.15.98:0/3449927471 shutdown_connections 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- 172.21.15.98:0/3449927471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c101820 0x7f079c1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- 172.21.15.98:0/3449927471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c0ffc90 0x7f079c1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- 172.21.15.98:0/3449927471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c100e80 0x7f079c1012e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- 172.21.15.98:0/3449927471 >> 172.21.15.98:0/3449927471 conn(0x7f079c0fb200 msgr2=0x7f079c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:03.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- 172.21.15.98:0/3449927471 shutdown_connections 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- 172.21.15.98:0/3449927471 wait complete. 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 Processor -- start 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 -- start start 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c100e80 0x7f079c196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.120+0000 7f07a3888700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c101820 0x7f079c19d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a3888700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c19d660 con 0x7f079c0ffc90 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a3888700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c19d7a0 con 0x7f079c101820 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a3888700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c19d930 con 0x7f079c100e80 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41792/0 (socket says 172.21.15.98:41792) 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 -- 172.21.15.98:0/396781966 learned_addr learned my addr 172.21.15.98:0/396781966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-18T20:33:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1e25700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c101820 0x7f079c19d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a0e23700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c100e80 0x7f079c196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 -- 172.21.15.98:0/396781966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c100e80 msgr2=0x7f079c196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c100e80 0x7f079c196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 -- 172.21.15.98:0/396781966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c101820 msgr2=0x7f079c19d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c101820 0x7f079c19d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 -- 172.21.15.98:0/396781966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980095f0 con 0x7f079c0ffc90 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a1624700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f078c00d980 tx=0x7f078c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00f890 con 0x7f079c0ffc90 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f078c00fe80 con 0x7f079c0ffc90 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c19dc10 con 0x7f079c0ffc90 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c019360 con 0x7f079c0ffc90 2024-03-18T20:33:03.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.121+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c19e120 con 0x7f079c0ffc90 2024-03-18T20:33:03.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.122+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c1071d0 con 0x7f079c0ffc90 2024-03-18T20:33:03.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.123+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f078c00f9f0 con 0x7f079c0ffc90 2024-03-18T20:33:03.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.124+0000 7f07927fc700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 0x7f07880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:33:03.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.124+0000 7f07a0e23700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 0x7f07880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:33:03.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.124+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9194+0+0 (secure 0 0 0) 0x7f078c0cc710 con 0x7f079c0ffc90 2024-03-18T20:33:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.124+0000 7f07a0e23700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 0x7f07880a3660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0798009990 tx=0x7f0798005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:33:03.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.128+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f078c095a80 con 0x7f079c0ffc90 2024-03-18T20:33:03.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:02 smithi169 ceph-mon[48529]: pgmap v477: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:33:03.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:02 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/1623443294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:02 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/116493124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:02 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/254436949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:02 smithi136 ceph-mon[48073]: pgmap v477: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T20:33:03.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:02 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/1623443294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:02 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/116493124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:02 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/254436949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T20:33:03.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.353+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f079c067f40 con 0x7f079c0ffc90 2024-03-18T20:33:03.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.353+0000 7f07927fc700 1 -- 172.21.15.98:0/396781966 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7f078c0990a0 con 0x7f079c0ffc90 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1287611061 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2681086384 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3671856112 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1417871753 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2768304991 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:03.358 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:03.359 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/479479747 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:03.359 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2082705134 2024-03-19T20:19:45.512552+0000 2024-03-18T20:33:03.359 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3238391768 2024-03-19T20:19:25.585401+0000 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 msgr2=0x7f07880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 0x7f07880a3660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0798009990 tx=0x7f0798005e80 comp rx=0 tx=0).stop 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 msgr2=0x7f079c195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f078c00d980 tx=0x7f078c00dc90 comp rx=0 tx=0).stop 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 shutdown_connections 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:6800/3114713517,v1:172.21.15.98:6801/3114713517] conn(0x7f07880a11a0 0x7f07880a3660 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f079c101820 0x7f079c19d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f079c0ffc90 0x7f079c195b30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 --2- 172.21.15.98:0/396781966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f079c100e80 0x7f079c196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:33:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.355+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 >> 172.21.15.98:0/396781966 conn(0x7f079c0fb200 msgr2=0x7f079c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:33:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.356+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 shutdown_connections 2024-03-18T20:33:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-18T20:33:03.356+0000 7f07a3888700 1 -- 172.21.15.98:0/396781966 wait complete. 2024-03-18T20:33:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-18T20:33:03.372 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-03-18T20:33:03.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:33:03.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:33:03.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[109730]: fuse finished with error 0 and tester_r 0 2024-03-18T20:33:03.417 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:33:04.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/3269986826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:04.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:03 smithi098 ceph-mon[43462]: from='client.? 172.21.15.98:0/396781966' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:04.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:03 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/3269986826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:04.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:03 smithi169 ceph-mon[48529]: from='client.? 172.21.15.98:0/396781966' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:04.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:03 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/3269986826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T20:33:04.242 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:03 smithi136 ceph-mon[48073]: from='client.? 172.21.15.98:0/396781966' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T20:33:05.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:04 smithi098 ceph-mon[43462]: pgmap v478: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:33:05.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:04 smithi169 ceph-mon[48529]: pgmap v478: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:33:05.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:04 smithi136 ceph-mon[48073]: pgmap v478: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T20:33:07.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:06 smithi098 ceph-mon[43462]: pgmap v479: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:33:07.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:06 smithi169 ceph-mon[48529]: pgmap v479: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:33:07.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:06 smithi136 ceph-mon[48073]: pgmap v479: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-18T20:33:09.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:08 smithi098 ceph-mon[43462]: pgmap v480: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-18T20:33:09.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:08 smithi169 ceph-mon[48529]: pgmap v480: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-18T20:33:09.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:08 smithi136 ceph-mon[48073]: pgmap v480: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-18T20:33:09.518 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:33:09.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:33:09.519 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:33:09.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:33:09.541 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-18T20:33:09.601 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:33:09.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:33:09.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:33:09.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:33:09.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-18T20:33:09.780 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T20:33:09.790 INFO:tasks.cephadm:Teardown begin 2024-03-18T20:33:09.790 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-18T20:33:09.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:33:09.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:33:09.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:33:09.900 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T20:33:09.900 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T20:33:09.918 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T20:33:09.938 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T20:33:09.960 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T20:33:09.961 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T20:33:09.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a 2024-03-18T20:33:10.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:09 smithi098 systemd[1]: Stopping Ceph mon.a for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:10.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a[43439]: 2024-03-18T20:33:10.196+0000 7fe0f62e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:10.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a[43439]: 2024-03-18T20:33:10.196+0000 7fe0f62e8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T20:33:10.713 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 18 20:33:10 smithi098 bash[452328]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-a 2024-03-18T20:33:10.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T20:33:10.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:10.764 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T20:33:10.764 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T20:33:10.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.b 2024-03-18T20:33:11.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:10 smithi136 systemd[1]: Stopping Ceph mon.b for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:11.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-b[48050]: 2024-03-18T20:33:11.002+0000 7f1721df5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:11.240 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-b[48050]: 2024-03-18T20:33:11.002+0000 7f1721df5700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T20:33:11.643 INFO:journalctl@ceph.mon.b.smithi136.stdout:Mar 18 20:33:11 smithi136 bash[88363]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-b 2024-03-18T20:33:11.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T20:33:11.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:11.712 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T20:33:11.712 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T20:33:11.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.c 2024-03-18T20:33:12.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:11 smithi169 systemd[1]: Stopping Ceph mon.c for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:12.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-c[48504]: 2024-03-18T20:33:11.951+0000 7f5f34fba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:12.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-c[48504]: 2024-03-18T20:33:11.951+0000 7f5f34fba700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T20:33:12.515 INFO:journalctl@ceph.mon.c.smithi169.stdout:Mar 18 20:33:12 smithi169 bash[90209]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mon-c 2024-03-18T20:33:12.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T20:33:12.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:12.577 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T20:33:12.577 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T20:33:12.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x 2024-03-18T20:33:12.939 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:33:12 smithi098 systemd[1]: Stopping Ceph mgr.x for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:12.939 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:33:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x[43714]: 2024-03-18T20:33:12.803+0000 7f4b1306f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:13.315 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 18 20:33:13 smithi098 bash[452467]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-x 2024-03-18T20:33:13.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T20:33:13.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:13.358 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T20:33:13.359 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T20:33:13.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.y 2024-03-18T20:33:13.740 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:33:13 smithi136 systemd[1]: Stopping Ceph mgr.y for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:14.080 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:33:13 smithi136 bash[88512]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-y 2024-03-18T20:33:14.080 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 18 20:33:13 smithi136 systemd[1]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-18T20:33:14.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T20:33:14.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:14.123 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T20:33:14.123 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T20:33:14.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.z 2024-03-18T20:33:14.484 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:33:14 smithi169 systemd[1]: Stopping Ceph mgr.z for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:14.836 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:33:14 smithi169 bash[90360]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mgr-z 2024-03-18T20:33:14.837 INFO:journalctl@ceph.mgr.z.smithi169.stdout:Mar 18 20:33:14 smithi169 systemd[1]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-18T20:33:14.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T20:33:14.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:14.880 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T20:33:14.880 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T20:33:14.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.0 2024-03-18T20:33:15.342 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:33:14 smithi098 systemd[1]: Stopping Ceph osd.0 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:15.342 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:33:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:33:15.111+0000 7f0818964700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:15.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:33:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:33:15.111+0000 7f0818964700 -1 osd.0 89 *** Got signal Terminated *** 2024-03-18T20:33:15.343 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:33:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0[62530]: 2024-03-18T20:33:15.111+0000 7f0818964700 -1 osd.0 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:21.093 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 18 20:33:20 smithi098 bash[452609]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-0 2024-03-18T20:33:21.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T20:33:21.855 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:21.856 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T20:33:21.856 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T20:33:21.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.1 2024-03-18T20:33:22.240 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:33:21 smithi136 systemd[1]: Stopping Ceph osd.1 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:22.240 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:33:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:33:22.099+0000 7fa4fec2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:22.240 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:33:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:33:22.099+0000 7fa4fec2e700 -1 osd.1 89 *** Got signal Terminated *** 2024-03-18T20:33:22.240 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:33:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1[53052]: 2024-03-18T20:33:22.099+0000 7fa4fec2e700 -1 osd.1 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:28.241 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 18 20:33:27 smithi136 bash[88652]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-1 2024-03-18T20:33:29.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T20:33:29.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:29.081 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T20:33:29.081 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T20:33:29.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.2 2024-03-18T20:33:29.383 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:33:29 smithi169 systemd[1]: Stopping Ceph osd.2 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:29.383 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:33:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:33:29.331+0000 7ff72b7aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:29.383 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:33:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:33:29.331+0000 7ff72b7aa700 -1 osd.2 89 *** Got signal Terminated *** 2024-03-18T20:33:29.383 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:33:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2[54341]: 2024-03-18T20:33:29.331+0000 7ff72b7aa700 -1 osd.2 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:35.383 INFO:journalctl@ceph.osd.2.smithi169.stdout:Mar 18 20:33:35 smithi169 bash[90502]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-2 2024-03-18T20:33:36.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T20:33:36.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:36.413 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T20:33:36.413 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T20:33:36.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.3 2024-03-18T20:33:36.842 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:33:36 smithi098 systemd[1]: Stopping Ceph osd.3 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:36.843 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:33:36 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:33:36.651+0000 7fbcb640f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:36.843 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:33:36 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:33:36.651+0000 7fbcb640f700 -1 osd.3 89 *** Got signal Terminated *** 2024-03-18T20:33:36.843 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:33:36 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3[71040]: 2024-03-18T20:33:36.651+0000 7fbcb640f700 -1 osd.3 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:41.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:41 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:41.199+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:41.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:41 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:41.673+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:42.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:41.814+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:42.593 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 18 20:33:42 smithi098 bash[452875]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-3 2024-03-18T20:33:42.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:42 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:42.245+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:42.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:42 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:42.642+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:43.232 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:42 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:42.848+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:43.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T20:33:43.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:43.616 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T20:33:43.616 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T20:33:43.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.4 2024-03-18T20:33:43.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:43 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:43.215+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:43.990 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:33:43 smithi136 systemd[1]: Stopping Ceph osd.4 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:43.991 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:33:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:33:43.845+0000 7f65c9e14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:43.991 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:33:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:33:43.845+0000 7f65c9e14700 -1 osd.4 89 *** Got signal Terminated *** 2024-03-18T20:33:43.991 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:33:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4[61736]: 2024-03-18T20:33:43.845+0000 7f65c9e14700 -1 osd.4 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:43.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:43 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:43.676+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:44.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:43 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:43.819+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:44.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:44 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:44.185+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:44.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:44 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:44.113+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:44.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:44 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:44.656+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:45.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:44 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:44.790+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:45.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:44 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:44.712+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:45.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:45 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:45.196+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:45.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:45 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:45.097+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:45.883 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:45 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:45.438+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:45.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:45 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:45.615+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:46.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:45 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:45.741+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:46.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:45 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:45.815+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:46.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:46 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:46.157+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:46.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:46 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:46.127+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:46.883 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:46 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:46.424+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:46.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:46 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:46.634+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:47.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:46 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:46.731+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:47.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:46 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:46.849+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:47.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:47 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:47.192+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:47.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:47 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:47.096+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:47.883 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:47 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:47.380+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:47.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:47 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:47.624+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:48.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:47 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:47.776+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:48.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:47 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:47.887+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:48.350 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:48 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:48.189+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:48.351 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:48 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:48.095+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:48.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:48 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:48.348+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:48.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:48 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:48.805+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:48.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:48 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:48.577+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:49.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:48 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:48.749+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:49.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:48 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:48.890+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:49.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:49 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:49.328+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:49.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:49 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:49.135+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:49.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:49 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:49.172+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:49.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:49 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:49.172+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:49.740 INFO:journalctl@ceph.osd.4.smithi136.stdout:Mar 18 20:33:49 smithi136 bash[88914]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-4 2024-03-18T20:33:49.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:49 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:49.561+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:50.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:49 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:49.709+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:50.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:49 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:49.893+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:50.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:49 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:49.893+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:50.102 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:49 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:49.824+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:50.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:50 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:50.329+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:50.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:50 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:50.164+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:50.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:50 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:50.164+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:50.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:50 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:50.158+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:50.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:50 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:50.603+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:50.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:50 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:50.603+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:50.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T20:33:50.818 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:50.818 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T20:33:50.818 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T20:33:50.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.5 2024-03-18T20:33:51.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:50.677+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:51.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:50.677+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:51.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:50.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:51.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:50.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:51.133 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:50 smithi169 systemd[1]: Stopping Ceph osd.5 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:51.133 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:51.070+0000 7f23cb3a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:51.133 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:51.070+0000 7f23cb3a2700 -1 osd.5 89 *** Got signal Terminated *** 2024-03-18T20:33:51.133 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:51.070+0000 7f23cb3a2700 -1 osd.5 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:51.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:50 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:50.813+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:51.312+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:51.312+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.148841+0000 front 2024-03-18T20:33:25.148767+0000 (oldest deadline 2024-03-18T20:33:50.448609+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:51.153+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:51.153+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:51.204+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:51.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:51 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:51.204+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:51.949 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:51 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:51.641+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:51.949 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:51 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:51.641+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:51.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:51 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:51.788+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:51.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:51 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:51.598+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:51.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:51 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:51.598+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:52.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:51 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:51.946+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:52.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:51 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:51.946+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:52.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:52.281+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:52.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:52.281+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.148841+0000 front 2024-03-18T20:33:25.148767+0000 (oldest deadline 2024-03-18T20:33:50.448609+0000) 2024-03-18T20:33:52.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:52.151+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:52.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:52.151+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:52.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:52.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:52.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:52 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:52.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:52.916 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:52.662+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:52.916 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:52.662+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:52.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:52 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:52.803+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:52.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:52 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:52.630+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:52.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:52 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:52.630+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:53.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:52.912+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:53.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:52.912+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:53.265+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:53.265+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.148841+0000 front 2024-03-18T20:33:25.148767+0000 (oldest deadline 2024-03-18T20:33:50.448609+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:53.254+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:53.254+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:53.103+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:53.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:53 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:53.103+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:53.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:53 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:53.782+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:53.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:53 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:53.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:53.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:53 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:53.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:54.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:53 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:53.702+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:54.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:53 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:53.702+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:54.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:53 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:53.904+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:54.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:53 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:53.904+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:54.384 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:54.063+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:54.384 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:54.063+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:54.384 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:54.279+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:54.384 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:54.280+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.148841+0000 front 2024-03-18T20:33:25.148767+0000 (oldest deadline 2024-03-18T20:33:50.448609+0000) 2024-03-18T20:33:54.385 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:54.278+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:54.385 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:54 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:54.278+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:54.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:54 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:54.793+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:54.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:54 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:54.640+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:54.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:54 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:54.640+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:55.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:54 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:54.659+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:55.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:54 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:54.659+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:55.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:54 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:54.865+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:55.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:54 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:54.865+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:55.383 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:55.310+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:19.248314+0000 front 2024-03-18T20:33:19.248150+0000 (oldest deadline 2024-03-18T20:33:45.148099+0000) 2024-03-18T20:33:55.383 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5[62167]: 2024-03-18T20:33:55.310+0000 7f23c6195700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.148841+0000 front 2024-03-18T20:33:25.148767+0000 (oldest deadline 2024-03-18T20:33:50.448609+0000) 2024-03-18T20:33:55.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:55.093+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:55.384 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:55.093+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:55.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:55.253+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:55.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:55 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:55.253+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:55.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:55 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:55.754+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:55.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:55 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:55.754+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:33:55.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:55 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:55.661+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:55.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:55 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:55.661+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:56.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:55.640+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:56.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:55.640+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:56.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:55.836+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:56.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:55.836+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:56.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:55 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:55.836+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:33:56.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:56 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:56.106+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:56.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:56 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:56.106+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:56.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:56 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:56.256+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:56.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:56 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:56.256+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:56.883 INFO:journalctl@ceph.osd.5.smithi169.stdout:Mar 18 20:33:56 smithi169 bash[90764]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-5 2024-03-18T20:33:56.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:56.753+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:56.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:56.753+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:33:56.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:56.654+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:56.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:56.654+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:56.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:56.654+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:33:57.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:56.670+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:57.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:56.670+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:57.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:56.670+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:33:57.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:56.801+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:57.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:56.801+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:57.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:56 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:56.801+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:33:57.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:57 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:57.217+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:57.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:57 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:57.217+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:57.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:57 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:57.126+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:57.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:57 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:57.126+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:57.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T20:33:57.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:57 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:57.793+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:57.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:57 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:57.793+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:33:57.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:57 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:57.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:57.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:57 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:57.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:57.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:57 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:57.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:33:58.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:33:58.009 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T20:33:58.010 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T20:33:58.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.6 2024-03-18T20:33:58.041 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:57.700+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:58.041 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:57.700+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:58.041 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:57.700+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:33:58.042 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:57.794+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:58.042 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:57.794+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:58.042 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:57 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:57.794+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:33:58.267 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:58 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:58.158+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:58.267 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:58 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:58.158+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:58.336 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 systemd[1]: Stopping Ceph osd.6 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:33:58.337 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.333+0000 7f391e13c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:33:58.592 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.333+0000 7f391e13c700 -1 osd.6 89 *** Got signal Terminated *** 2024-03-18T20:33:58.592 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.333+0000 7f391e13c700 -1 osd.6 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:33:58.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:58 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:58.262+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:58.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:58 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:58.262+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:58.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:58.813+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:58.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:58.813+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:33:58.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:58.691+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:58.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:58.691+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:58.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:58 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:58.691+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:33:59.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.742+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:33:59.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.742+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:33:59.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:58.742+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:33:59.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:58.824+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:33:59.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:58.824+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:33:59.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:58 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:58.824+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:33:59.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:59 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:59.208+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:33:59.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:33:59 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:33:59.208+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:33:59.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:59 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:59.245+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:33:59.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:33:59 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:33:59.245+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:33:59.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:59 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:59.821+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:33:59.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:33:59 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:33:59.821+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:33:59.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:59 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:59.714+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:33:59.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:59 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:59.714+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:33:59.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:33:59 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:33:59.714+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:00.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:59.771+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:34:00.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:59.771+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:34:00.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:33:59.771+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:34:00.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:59.806+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:00.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:59.806+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:00.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:33:59 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:33:59.806+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:00.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:00 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:00.193+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:00.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:00 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:00.193+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:00.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:00 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:00.276+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:00.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:00 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:00.276+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:00.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:00 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:00.276+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:00.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:00 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:00.807+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:00.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:00 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:00.807+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:00.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:00 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:00.665+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:00.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:00 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:00.665+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:00.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:00 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:00.665+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:01.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:00.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:34:01.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:00.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:34:01.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:00.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:34:01.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:00.813+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:01.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:00.813+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:01.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:00 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:00.813+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:01.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:01 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:01.174+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:01.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:01 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:01.174+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:01.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:01 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:01.307+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:01.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:01 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:01.307+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:01.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:01 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:01.307+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:01.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:01 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:01.785+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:01.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:01 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:01.785+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:01.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:01 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:01.699+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:01.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:01 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:01.699+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:01.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:01 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:01.699+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:02.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:01.804+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:02.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:01.804+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:02.092 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:01.804+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:02.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:01.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:34:02.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:01.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:34:02.093 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:01 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:01.764+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:34:02.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:02 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:02.199+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:02.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:02 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:02.199+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:02.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:02 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:02.329+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:02.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:02 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:02.329+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:02.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:02 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:02.329+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:02.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:02.816+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:02.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:02.816+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:02.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:02.816+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:02.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:02.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:02.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:02.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:02.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:02 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:02.677+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:03.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:02.787+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.004018+0000 front 2024-03-18T20:33:18.004153+0000 (oldest deadline 2024-03-18T20:33:43.904247+0000) 2024-03-18T20:34:03.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:02.787+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:23.904760+0000 front 2024-03-18T20:33:23.904844+0000 (oldest deadline 2024-03-18T20:33:49.804888+0000) 2024-03-18T20:34:03.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6[79115]: 2024-03-18T20:34:02.787+0000 7f3918f2f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:29.805490+0000 front 2024-03-18T20:33:29.805617+0000 (oldest deadline 2024-03-18T20:33:55.705407+0000) 2024-03-18T20:34:03.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:02.787+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:03.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:02.787+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:03.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:02 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:02.787+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:03.189+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:03.189+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:03.189+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:03.373+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:03.373+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:03.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:03 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:03.373+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:03.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:03.863+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:03.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:03.863+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:03.990 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:03.863+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:03.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:03.686+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:03.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:03.686+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:03.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:03 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:03.686+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:04.092 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 18 20:34:03 smithi098 bash[453136]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-6 2024-03-18T20:34:04.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:03.820+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:04.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:03.820+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:04.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:03.820+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:04.093 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:03 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:03.820+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:04.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:04.212+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:04.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:04.212+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:04.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:04.212+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:04.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:04.336+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:04.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:04.336+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:04.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:04 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:04.336+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:04.903 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:04.650+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:04.904 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:04.650+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:04.904 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:04.650+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:05.065 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:04.834+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:05.065 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:04.834+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:05.065 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:04.834+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:05.065 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:04 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:04.834+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:05.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T20:34:05.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:05.155 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T20:34:05.155 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T20:34:05.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.7 2024-03-18T20:34:05.185 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:04.901+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:05.185 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:04.901+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:05.185 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:04 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:04.901+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:05.490 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 systemd[1]: Stopping Ceph osd.7 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:05.490 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.398+0000 7efd2e3fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:05.490 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.398+0000 7efd2e3fd700 -1 osd.7 89 *** Got signal Terminated *** 2024-03-18T20:34:05.490 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.398+0000 7efd2e3fd700 -1 osd.7 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:05.214+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:05.214+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:05.214+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:05.325+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:05.325+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:05.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:05 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:05.325+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:05.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:05.602+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:05.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:05.602+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:05.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:05.602+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:06.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.910+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:06.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.910+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:06.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:05 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:05.910+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:06.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:05.850+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:06.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:05.850+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:06.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:05.850+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:06.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:05 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:05.850+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:06.198+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:06.198+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:06.198+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:06.352+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:06.352+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:06.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:06 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:06.352+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:06.911 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:06.638+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:06.912 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:06.638+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:06.912 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:06.638+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:06.912 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:06.638+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:07.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:06.908+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:07.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:06.908+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:07.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:06 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:06.908+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:07.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:06.898+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:07.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:06.898+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:07.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:06.898+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:07.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:06 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:06.898+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:07.227+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:07.227+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:07.227+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:07.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:07.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:07.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:07 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:07.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:07.886 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:07.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:07.886 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:07.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:07.886 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:07.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:07.886 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:07.608+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:08.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:07.882+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:08.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:07.882+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:08.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:07.882+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:08.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:07.886+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:08.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:07.886+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:08.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:07.886+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:08.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:07 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:07.886+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:08.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:08.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:08.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:08.299+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:08.299+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:08.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:08 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:08.299+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:08.931 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:08.571+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:08.931 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:08.571+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:08.931 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:08.571+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:08.931 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:08.571+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:09.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:08.927+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:09.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:08.927+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:09.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:08 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:08.927+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:09.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:08.911+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:09.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:08.911+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:09.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:08.911+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:09.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:08 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:08.911+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:09.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:09.197+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:09.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:09.197+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:09.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:09.197+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:09.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:09.320+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:09.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:09.320+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:09.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:09 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:09.320+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:09.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:09.546+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:09.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:09.546+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:09.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:09.546+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:09.913 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:09.546+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:10.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:09.909+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.331049+0000 front 2024-03-18T20:33:26.331033+0000 (oldest deadline 2024-03-18T20:33:48.631170+0000) 2024-03-18T20:34:10.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:09.909+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:32.732364+0000 front 2024-03-18T20:33:32.732329+0000 (oldest deadline 2024-03-18T20:33:55.032342+0000) 2024-03-18T20:34:10.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:09 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7[69794]: 2024-03-18T20:34:09.909+0000 7efd269eb700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:37.333246+0000 front 2024-03-18T20:33:37.333305+0000 (oldest deadline 2024-03-18T20:34:02.633257+0000) 2024-03-18T20:34:10.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:09.896+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:10.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:09.896+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:10.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:09.896+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:10.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:09 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:09.896+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:10.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:10.202+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:10.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:10.202+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:10.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:10.202+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:10.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:10.283+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:10.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:10.283+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:10.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:10.283+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:10.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:10 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:10.283+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:10.856 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:10 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:10.569+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:10.857 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:10 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:10.569+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:10.857 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:10 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:10.569+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:10.857 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:10 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:10.569+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:11.240 INFO:journalctl@ceph.osd.7.smithi136.stdout:Mar 18 20:34:10 smithi136 bash[89177]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-7 2024-03-18T20:34:11.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:10.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:11.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:10.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:11.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:10.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:11.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:10 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:10.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:11.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:11.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:11.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:11.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:11.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:11.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:11.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:11.219+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:11.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:11.327+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:11.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:11.327+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:11.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:11.327+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:11.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:11 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:11.327+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:11.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:11.521+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:11.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:11.521+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:11.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:11.521+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:11.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:11.521+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:11.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:11 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:11.521+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:12.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T20:34:12.312 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:12.312 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T20:34:12.313 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T20:34:12.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.8 2024-03-18T20:34:12.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:11 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:11.871+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:12.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:11 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:11.871+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:12.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:11 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:11.871+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:12.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:11 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:11.871+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:12.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:11 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:11.871+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:12.550 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:12.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:12.550 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:12.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:12.550 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:12.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:12.550 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:12.324+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.246+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.246+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.246+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.246+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 systemd[1]: Stopping Ceph osd.8 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:12.551 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.546+0000 7f371957a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:12.883 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.546+0000 7f371957a700 -1 osd.8 89 *** Got signal Terminated *** 2024-03-18T20:34:12.883 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:12 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:12.546+0000 7f371957a700 -1 osd.8 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:34:12.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:12.561+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:12.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:12.561+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:12.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:12.561+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:12.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:12.562+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:12.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:12.562+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:13.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:12.869+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:13.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:12.869+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:13.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:12.869+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:13.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:12.869+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:13.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:12 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:12.869+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:13.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:13.277+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:13.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:13.277+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:13.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:13.277+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:13.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:13.277+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:13.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:13.298+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:13.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:13.298+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:13.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:13.298+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:13.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:13 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:13.298+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:13.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:13.531+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:14.342 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:13.531+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:14.342 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:13.531+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:14.342 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:13.531+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:14.342 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:13 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:13.531+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:14.344 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:13 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:13.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:14.344 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:13 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:13.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:14.345 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:13 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:13.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:14.345 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:13 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:13.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:14.345 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:13 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:13.863+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:14.254+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:14.254+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:14.254+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:14.254+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:14.313+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:14.313+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:14.313+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:14.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:14 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:14.313+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:14.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:14 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:14.500+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:14.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:14 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:14.500+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:14.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:14 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:14.500+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:14.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:14 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:14.500+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:14.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:14 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:14.500+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:15.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:14 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:14.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:15.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:14 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:14.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:15.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:14 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:14.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:15.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:14 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:14.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:15.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:14 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:14.900+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:15.269+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:15.269+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:15.269+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:15.269+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:15.271+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:15.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:15.271+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:15.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:15.271+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:15.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:15 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:15.271+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:15.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:15 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:15.461+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:15.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:15 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:15.461+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:15.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:15 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:15.461+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:15.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:15 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:15.461+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:15.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:15 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:15.461+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:16.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:15.934+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:16.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:15.934+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:16.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:15.934+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:16.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:15.934+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:16.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:15 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:15.934+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:16.261+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:16.261+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:16.261+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:16.261+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:16.285+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:16.285+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:16.633 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:16.285+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:16.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:16 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:16.285+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:16.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:16 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:16.469+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:16.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:16 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:16.469+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:16.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:16 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:16.469+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:16.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:16 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:16.469+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:16.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:16 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:16.469+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:17.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:16 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:16.922+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:17.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:16 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:16.922+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:17.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:16 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:16.922+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:17.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:16 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:16.922+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:17.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:16 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:16.922+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:17.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:17.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:17.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:17.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:17.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:17.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:17.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:17.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:17.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:17.330+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:16.267188+0000 front 2024-03-18T20:33:16.267004+0000 (oldest deadline 2024-03-18T20:33:40.967046+0000) 2024-03-18T20:34:17.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:17.330+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.968684+0000 front 2024-03-18T20:33:24.968836+0000 (oldest deadline 2024-03-18T20:33:48.468740+0000) 2024-03-18T20:34:17.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:17.330+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.072041+0000 front 2024-03-18T20:33:40.072066+0000 (oldest deadline 2024-03-18T20:34:02.371993+0000) 2024-03-18T20:34:17.634 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:17 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8[70210]: 2024-03-18T20:34:17.330+0000 7f371336b700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.273281+0000 front 2024-03-18T20:33:45.273234+0000 (oldest deadline 2024-03-18T20:34:10.573250+0000) 2024-03-18T20:34:17.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:17.423+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:17.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:17.423+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:17.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:17.423+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:17.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:17.423+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:17.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:17.423+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:18.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:17 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:17.884+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:18.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:17 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:17.884+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:18.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:17 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:17.884+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:18.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:17 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:17.884+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:18.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:17 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:17.884+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:18.383 INFO:journalctl@ceph.osd.8.smithi169.stdout:Mar 18 20:34:18 smithi169 bash[91018]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-8 2024-03-18T20:34:18.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:18 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:18.268+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:18.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:18 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:18.268+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:18.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:18 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:18.268+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:18.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:18 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:18.268+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:18.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:18.429+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:18.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:18.429+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:18.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:18.429+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:18.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:18.429+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:18.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:18 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:18.429+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:19.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:18 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:18.931+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:19.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:19 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:19.247+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:19.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:19 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:19.247+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:19.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:19 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:19.247+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:19.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:19 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:19.247+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:19.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:19 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:19.247+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:19.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T20:34:19.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:19.501 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T20:34:19.502 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T20:34:19.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.9 2024-03-18T20:34:19.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:19.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:19.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:19.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:19.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:19.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:19 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:19.467+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:19.842 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 systemd[1]: Stopping Ceph osd.9 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:19.842 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.745+0000 7fede868a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:19.843 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.745+0000 7fede868a700 -1 osd.9 89 *** Got signal Terminated *** 2024-03-18T20:34:19.843 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.745+0000 7fede868a700 -1 osd.9 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:34:20.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:20.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:20.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:20.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:20.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:20.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:19 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:19.923+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:20.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:20 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:20.295+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:20.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:20 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:20.295+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:20.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:20 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:20.295+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:20.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:20 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:20.295+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:20.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:20 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:20.295+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:20.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:20 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:20.420+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:21.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:21.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:21.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:21.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:21.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:21.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:20 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:20.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:21.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:21.288+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:21.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:21.288+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:21.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:21.288+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:21.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:21.288+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:21.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:21 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:21.288+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:21.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:21 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:21.426+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:22.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:21 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:21.942+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:22.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:22.274+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:22.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:22.274+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:22.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:22.274+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:22.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:22.274+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:22.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:22.274+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:22.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:22 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:22.435+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:23.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:22 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:22.947+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:23.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:23.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:23.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:23.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:23.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:23.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:23.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:23.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:23.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:23 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:23.231+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:23.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:23.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:23.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:23.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:23.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:23.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:23 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:23.448+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:24.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.042741+0000 front 2024-03-18T20:33:17.042718+0000 (oldest deadline 2024-03-18T20:33:41.742893+0000) 2024-03-18T20:34:24.342 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:26.344515+0000 front 2024-03-18T20:33:26.344646+0000 (oldest deadline 2024-03-18T20:33:49.844594+0000) 2024-03-18T20:34:24.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:33.345817+0000 front 2024-03-18T20:33:33.345844+0000 (oldest deadline 2024-03-18T20:33:55.546543+0000) 2024-03-18T20:34:24.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:41.447342+0000 front 2024-03-18T20:33:41.447387+0000 (oldest deadline 2024-03-18T20:34:03.147516+0000) 2024-03-18T20:34:24.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:47.748977+0000 front 2024-03-18T20:33:47.749063+0000 (oldest deadline 2024-03-18T20:34:11.848967+0000) 2024-03-18T20:34:24.343 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:23 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9[87101]: 2024-03-18T20:34:23.919+0000 7fede4c80700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.550062+0000 front 2024-03-18T20:33:53.550101+0000 (oldest deadline 2024-03-18T20:34:18.250068+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:24.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:24 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:24.258+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:24.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:24.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:24.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:24.742 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:24.742 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:24.742 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:24 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:24.407+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:25.592 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 18 20:34:25 smithi098 bash[453393]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-9 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:25.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:25 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:25.251+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:25.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:25 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:25.368+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:26.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T20:34:26.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:26.476 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T20:34:26.476 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T20:34:26.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.10 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:26.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:26 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:26.232+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:26.714 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:26.714 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:26.714 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:26.714 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:26.714 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:26.715 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:26.717 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.387+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:26.717 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 systemd[1]: Stopping Ceph osd.10 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:26.717 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.710+0000 7fa763e4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:26.717 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.710+0000 7fa763e4e700 -1 osd.10 89 *** Got signal Terminated *** 2024-03-18T20:34:26.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:26 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:26.710+0000 7fa763e4e700 -1 osd.10 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:27.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:27.206+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:27.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:27 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:27.437+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:28.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:28 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:28.255+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:28.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:28.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:28.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:28.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:28.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:28.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:28.741 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:28 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:28.473+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:29.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:29 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:29.219+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:29.740 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:29 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:29.454+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:30.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:30.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:30.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:30.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:30.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:30.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:30.182+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:30.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:30.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:30.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:30.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:30.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:30.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:30.991 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:30 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:30.501+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:31.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:31 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:31.146+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:17.203841+0000 front 2024-03-18T20:33:17.203875+0000 (oldest deadline 2024-03-18T20:33:40.703863+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:24.705370+0000 front 2024-03-18T20:33:24.705449+0000 (oldest deadline 2024-03-18T20:33:50.005567+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:31.007170+0000 front 2024-03-18T20:33:31.007187+0000 (oldest deadline 2024-03-18T20:33:56.207704+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-18T20:33:40.309040+0000 front 2024-03-18T20:33:40.308928+0000 (oldest deadline 2024-03-18T20:34:05.608926+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:45.609320+0000 front 2024-03-18T20:33:45.609367+0000 (oldest deadline 2024-03-18T20:34:10.909389+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:55.010371+0000 front 2024-03-18T20:33:55.010388+0000 (oldest deadline 2024-03-18T20:34:18.510189+0000) 2024-03-18T20:34:31.990 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:31 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10[77793]: 2024-03-18T20:34:31.535+0000 7fa75c43c700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:01.411446+0000 front 2024-03-18T20:34:01.411504+0000 (oldest deadline 2024-03-18T20:34:24.911302+0000) 2024-03-18T20:34:32.490 INFO:journalctl@ceph.osd.10.smithi136.stdout:Mar 18 20:34:32 smithi136 bash[89436]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-10 2024-03-18T20:34:32.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:32.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:32.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:32.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:32.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:32.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:32.634 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:32 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:32.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:33.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T20:34:33.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:33.625 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T20:34:33.626 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T20:34:33.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.11 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:33.633 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.151+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 systemd[1]: Stopping Ceph osd.11 for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.862+0000 7fb2d16dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.862+0000 7fb2d16dd700 -1 osd.11 89 *** Got signal Terminated *** 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:33 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:33.862+0000 7fb2d16dd700 -1 osd.11 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:34.110 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:34.382 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:34 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:34.105+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:35.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:35.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:35.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:35.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:35.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:35.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:35.384 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:35 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:35.077+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:36.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:36.034+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:37.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:37 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:37.026+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-18T20:33:18.336733+0000 front 2024-03-18T20:33:18.336659+0000 (oldest deadline 2024-03-18T20:33:43.636732+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2024-03-18T20:33:25.338096+0000 front 2024-03-18T20:33:25.338054+0000 (oldest deadline 2024-03-18T20:33:50.638157+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6804 osd.2 since back 2024-03-18T20:33:30.638546+0000 front 2024-03-18T20:33:30.638509+0000 (oldest deadline 2024-03-18T20:33:59.439414+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6812 osd.4 since back 2024-03-18T20:33:48.741613+0000 front 2024-03-18T20:33:48.741678+0000 (oldest deadline 2024-03-18T20:34:09.742154+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.169:6812 osd.5 since back 2024-03-18T20:33:53.243032+0000 front 2024-03-18T20:33:53.243158+0000 (oldest deadline 2024-03-18T20:34:19.143155+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-18T20:34:02.644264+0000 front 2024-03-18T20:34:02.644220+0000 (oldest deadline 2024-03-18T20:34:23.744147+0000) 2024-03-18T20:34:38.383 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:38 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11[79008]: 2024-03-18T20:34:38.051+0000 7fb2ca4cc700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.136:6820 osd.7 since back 2024-03-18T20:34:07.244866+0000 front 2024-03-18T20:34:07.244943+0000 (oldest deadline 2024-03-18T20:34:31.944924+0000) 2024-03-18T20:34:39.882 INFO:journalctl@ceph.osd.11.smithi169.stdout:Mar 18 20:34:39 smithi169 bash[91270]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-osd-11 2024-03-18T20:34:40.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T20:34:40.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:40.877 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T20:34:40.877 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T20:34:40.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.a 2024-03-18T20:34:41.342 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:34:40 smithi098 systemd[1]: Stopping Ceph mds.a for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:41.342 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:34:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-a[94819]: 2024-03-18T20:34:41.150+0000 7fc06f953700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:41.342 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:34:41 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-a[94819]: 2024-03-18T20:34:41.150+0000 7fc06f953700 -1 mds.a *** got signal Terminated *** 2024-03-18T20:34:45.699 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 18 20:34:45 smithi098 bash[453652]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-a 2024-03-18T20:34:45.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T20:34:45.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:45.842 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T20:34:45.842 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T20:34:45.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.d 2024-03-18T20:34:46.092 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:34:45 smithi098 systemd[1]: Stopping Ceph mds.d for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:46.592 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:34:46 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-d[93909]: 2024-03-18T20:34:46.117+0000 7f2c87459700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:46.592 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:34:46 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-d[93909]: 2024-03-18T20:34:46.117+0000 7f2c87459700 -1 mds.d *** got signal Terminated *** 2024-03-18T20:34:49.824 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 18 20:34:49 smithi098 bash[453799]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-d 2024-03-18T20:34:49.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T20:34:49.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:49.907 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T20:34:49.907 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T20:34:49.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.g 2024-03-18T20:34:50.092 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:34:49 smithi098 systemd[1]: Stopping Ceph mds.g for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:50.592 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:34:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-g[94430]: 2024-03-18T20:34:50.149+0000 7f7aea09a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:50.592 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:34:50 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-g[94430]: 2024-03-18T20:34:50.149+0000 7f7aea09a700 -1 mds.g *** got signal Terminated *** 2024-03-18T20:34:52.693 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 18 20:34:52 smithi098 bash[453938]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-g 2024-03-18T20:34:52.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T20:34:52.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:52.734 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T20:34:52.734 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T20:34:52.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.j 2024-03-18T20:34:52.994 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:34:52 smithi098 systemd[1]: Stopping Ceph mds.j for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:53.342 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:34:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-j[93524]: 2024-03-18T20:34:52.990+0000 7f1c6db4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:53.342 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:34:52 smithi098 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-j[93524]: 2024-03-18T20:34:52.990+0000 7f1c6db4d700 -1 mds.j *** got signal Terminated *** 2024-03-18T20:34:56.250 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 18 20:34:55 smithi098 bash[454078]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-j 2024-03-18T20:34:56.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T20:34:56.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:34:56.288 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T20:34:56.288 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T20:34:56.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.b 2024-03-18T20:34:56.740 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:34:56 smithi136 systemd[1]: Stopping Ceph mds.b for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:34:56.740 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:34:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-b[84669]: 2024-03-18T20:34:56.536+0000 7f98bbed4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:34:56.740 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:34:56 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-b[84669]: 2024-03-18T20:34:56.536+0000 7f98bbed4700 -1 mds.b *** got signal Terminated *** 2024-03-18T20:35:06.991 INFO:journalctl@ceph.mds.b.smithi136.stdout:Mar 18 20:35:06 smithi136 bash[89699]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-b 2024-03-18T20:35:07.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T20:35:07.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:07.137 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T20:35:07.138 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T20:35:07.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.e 2024-03-18T20:35:07.414 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:35:07 smithi136 systemd[1]: Stopping Ceph mds.e for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:07.740 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:35:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-e[83983]: 2024-03-18T20:35:07.410+0000 7f9b0ba46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:07.740 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:35:07 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-e[83983]: 2024-03-18T20:35:07.410+0000 7f9b0ba46700 -1 mds.e *** got signal Terminated *** 2024-03-18T20:35:11.740 INFO:journalctl@ceph.mds.e.smithi136.stdout:Mar 18 20:35:11 smithi136 bash[89839]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-e 2024-03-18T20:35:11.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T20:35:11.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:11.904 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T20:35:11.904 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T20:35:11.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.h 2024-03-18T20:35:12.172 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:35:11 smithi136 systemd[1]: Stopping Ceph mds.h for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:12.490 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:35:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-h[84327]: 2024-03-18T20:35:12.169+0000 7f80c9ab5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:12.490 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:35:12 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-h[84327]: 2024-03-18T20:35:12.169+0000 7f80c9ab5700 -1 mds.h *** got signal Terminated *** 2024-03-18T20:35:16.990 INFO:journalctl@ceph.mds.h.smithi136.stdout:Mar 18 20:35:16 smithi136 bash[89977]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-h 2024-03-18T20:35:17.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T20:35:17.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:17.077 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T20:35:17.078 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T20:35:17.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.k 2024-03-18T20:35:17.344 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:35:17 smithi136 systemd[1]: Stopping Ceph mds.k for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:17.740 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:35:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-k[83641]: 2024-03-18T20:35:17.340+0000 7f1f01221700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:17.740 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:35:17 smithi136 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-k[83641]: 2024-03-18T20:35:17.340+0000 7f1f01221700 -1 mds.k *** got signal Terminated *** 2024-03-18T20:35:21.740 INFO:journalctl@ceph.mds.k.smithi136.stdout:Mar 18 20:35:21 smithi136 bash[90116]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-k 2024-03-18T20:35:21.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T20:35:21.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:21.962 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T20:35:21.963 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T20:35:21.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.c 2024-03-18T20:35:22.383 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:35:22 smithi169 systemd[1]: Stopping Ceph mds.c for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:22.383 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:35:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-c[85851]: 2024-03-18T20:35:22.205+0000 7f25b839b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:22.383 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:35:22 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-c[85851]: 2024-03-18T20:35:22.205+0000 7f25b839b700 -1 mds.c *** got signal Terminated *** 2024-03-18T20:35:26.632 INFO:journalctl@ceph.mds.c.smithi169.stdout:Mar 18 20:35:26 smithi169 bash[91534]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-c 2024-03-18T20:35:26.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T20:35:26.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:26.762 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T20:35:26.763 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T20:35:26.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.f 2024-03-18T20:35:27.049 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:35:26 smithi169 systemd[1]: Stopping Ceph mds.f for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:27.382 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:35:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-f[85509]: 2024-03-18T20:35:27.044+0000 7f3cb47d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:27.383 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:35:27 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-f[85509]: 2024-03-18T20:35:27.044+0000 7f3cb47d1700 -1 mds.f *** got signal Terminated *** 2024-03-18T20:35:29.836 INFO:journalctl@ceph.mds.f.smithi169.stdout:Mar 18 20:35:29 smithi169 bash[91672]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-f 2024-03-18T20:35:29.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T20:35:29.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:29.877 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T20:35:29.877 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T20:35:29.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.i 2024-03-18T20:35:30.133 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:35:29 smithi169 systemd[1]: Stopping Ceph mds.i for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:30.633 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:35:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-i[86191]: 2024-03-18T20:35:30.137+0000 7fa37673b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:30.633 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:35:30 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-i[86191]: 2024-03-18T20:35:30.137+0000 7fa37673b700 -1 mds.i *** got signal Terminated *** 2024-03-18T20:35:36.086 INFO:journalctl@ceph.mds.i.smithi169.stdout:Mar 18 20:35:35 smithi169 bash[91813]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-i 2024-03-18T20:35:36.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T20:35:36.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:36.124 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T20:35:36.124 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T20:35:36.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.l 2024-03-18T20:35:36.382 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:35:36 smithi169 systemd[1]: Stopping Ceph mds.l for a6e2b072-e564-11ee-95c9-87774f69a715... 2024-03-18T20:35:36.883 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:35:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-l[85167]: 2024-03-18T20:35:36.385+0000 7f07951a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T20:35:36.883 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:35:36 smithi169 ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-l[85167]: 2024-03-18T20:35:36.385+0000 7f07951a4700 -1 mds.l *** got signal Terminated *** 2024-03-18T20:35:40.883 INFO:journalctl@ceph.mds.l.smithi169.stdout:Mar 18 20:35:40 smithi169 bash[91951]: ceph-a6e2b072-e564-11ee-95c9-87774f69a715-mds-l 2024-03-18T20:35:40.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a6e2b072-e564-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T20:35:40.984 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T20:35:40.984 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T20:35:40.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T20:35:41.168 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:35:42.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T20:35:42.731 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:35:44.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T20:35:44.314 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:35:45.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:35:45.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:35:45.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T20:35:45.832 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T20:35:45.834 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi098/crash 2024-03-18T20:35:45.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash -- . 2024-03-18T20:35:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T20:35:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-18T20:35:45.867 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi136/crash 2024-03-18T20:35:45.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash -- . 2024-03-18T20:35:45.923 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T20:35:45.923 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-03-18T20:35:45.925 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi169/crash 2024-03-18T20:35:45.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash -- . 2024-03-18T20:35:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T20:35:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-03-18T20:35:45.966 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T20:35:45.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-18T20:35:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:grep: /var/log/ceph/a6e2b072-e564-11ee-95c9-87774f69a715/ceph.log: No such file or directory 2024-03-18T20:35:46.003 INFO:tasks.cephadm:Compressing logs... 2024-03-18T20:35:46.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:35:46.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:35:46.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:35:46.072 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T20:35:46.075 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T20:35:46.078 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T20:38:55.160 INFO:tasks.cephadm:Archiving logs... 2024-03-18T20:38:55.161 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi098/log 2024-03-18T20:38:55.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T20:39:18.925 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi136/log 2024-03-18T20:39:18.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T20:39:31.101 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi169/log 2024-03-18T20:39:31.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T20:39:34.472 INFO:tasks.cephadm:Removing cluster... 2024-03-18T20:39:34.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force 2024-03-18T20:39:34.659 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:39:34.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force 2024-03-18T20:39:35.096 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:39:35.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a6e2b072-e564-11ee-95c9-87774f69a715 --force 2024-03-18T20:39:35.610 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: a6e2b072-e564-11ee-95c9-87774f69a715 2024-03-18T20:39:35.874 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T20:39:35.874 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T20:39:35.896 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T20:39:35.915 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T20:39:35.933 INFO:tasks.cephadm:Teardown complete 2024-03-18T20:39:35.933 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T20:39:35.944 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-18T20:39:35.945 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T20:39:35.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T20:39:35.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T20:39:35.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T20:39:36.072 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T20:39:36.083 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T20:39:36.083 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:39:36.086 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:39:36.088 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:39:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2024-03-18T20:39:36.103 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2024-03-18T20:39:36.104 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +4558us[+4558us] +/- 117ms 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -2652us[-2631us] +/- 82ms 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -453us[ -453us] +/- 88ms 2024-03-18T20:39:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:39:36.109 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:39:36.109 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-18T20:39:36.109 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +4541us[+4541us] +/- 117ms 2024-03-18T20:39:36.109 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -2653us[-2682us] +/- 82ms 2024-03-18T20:39:36.109 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 -464us[ -464us] +/- 88ms 2024-03-18T20:39:36.110 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +4536us[+4536us] +/- 117ms 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -2666us[-2684us] +/- 82ms 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -466us[ -483us] +/- 88ms 2024-03-18T20:39:36.111 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:39:36.111 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T20:39:36.121 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T20:39:36.122 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T20:39:36.133 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T20:39:36.225 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T20:39:36.294 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T20:39:36.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:39:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:36.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:39:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:36.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:39:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:36.452 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T20:39:36.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T20:39:36.489 INFO:teuthology.task.internal:Duration was 2098.727729 seconds 2024-03-18T20:39:36.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T20:39:36.508 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T20:39:36.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T20:39:36.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T20:39:36.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T20:39:36.556 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:39:36.559 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:39:36.560 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:39:37.003 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T20:39:37.003 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-18T20:39:37.003 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T20:39:37.026 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-18T20:39:37.027 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T20:39:37.049 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-18T20:39:37.050 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T20:39:37.073 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T20:39:37.073 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:39:37.076 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:39:37.092 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:39:37.166 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T20:39:37.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T20:39:37.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T20:39:37.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T20:39:37.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T20:39:37.283 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T20:39:37.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T20:39:37.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T20:39:37.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T20:39:37.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T20:39:37.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.402 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-18T20:39:37.404 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-18T20:39:37.431 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-18T20:39:37.476 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:37.493 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:37.509 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:39:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:39:37.533 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T20:39:37.545 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T20:39:37.546 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi098 2024-03-18T20:39:37.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T20:39:37.589 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi136 2024-03-18T20:39:37.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T20:39:37.629 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371/remote/smithi169 2024-03-18T20:39:37.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T20:39:37.670 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T20:39:37.671 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T20:39:37.673 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T20:39:37.675 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T20:39:37.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T20:39:37.735 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T20:39:37.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T20:39:37.747 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T20:39:37.747 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T20:39:37.749 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T20:39:37.751 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T20:39:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 20:39 /home/ubuntu/cephtest 2024-03-18T20:39:37.766 INFO:teuthology.orchestra.run.smithi136.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 20:39 /home/ubuntu/cephtest 2024-03-18T20:39:37.777 INFO:teuthology.orchestra.run.smithi169.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 20:39 /home/ubuntu/cephtest 2024-03-18T20:39:37.778 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T20:39:37.791 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T20:39:37.805 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-18T20:39:37.805 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T20:39:37.829 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-18T20:39:37.830 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:39:37.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.978053', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIB8Ae+54B+tu+xrN4dy+6qA+Qfruw1AuNLlq7FRd96Jl8JDmjZYFUkn5qZsyajlFhIvrgjmUSUTkATTFEP91cE='} 2024-03-18T20:39:37.857 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-18T20:39:37.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:39:37.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.976381', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRDkwlMwjYLvjLrlv9nI3JnLihrK0ebfciUKdDCjNKRZYmxSYhZKqGJ4IFBpx2o1k4RzL5iGNDc94HW58lVrSI='} 2024-03-18T20:39:37.884 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-18T20:39:37.884 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:39:37.912 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-18T20:39:37.912 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T20:39:37.923 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 19:58:56.974485', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJO4obnKNnSNvDcGwYZnClympqs9IHaPi5bfFR+G2hsP/xbY7M+yPYH8Cbot4/MPEyNc2pVWtu5Orl4lxv+NTig='} 2024-03-18T20:39:37.935 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-18T20:39:37.935 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:37.955 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-18T20:39:37.955 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T20:39:37.968 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-18T20:39:37.978 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-18T20:39:37.978 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:37.995 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-18T20:39:37.996 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T20:39:38.005 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-18T20:39:38.019 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-18T20:39:38.019 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:38.042 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-18T20:39:41.969 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:42.006 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:42.043 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:42.080 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-18T20:39:42.116 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-18T20:39:42.153 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-18T20:39:46.082 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:46.117 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:46.154 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:46.193 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-18T20:39:46.226 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-18T20:39:46.263 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-18T20:39:46.364 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-18T20:39:50.196 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:50.227 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:39:50.307 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-18T20:39:50.337 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-18T20:39:50.408 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-18T20:39:50.438 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-18T20:39:50.543 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} duration: 2098.7277286052704 failure_reason: 'The following counters failed to be set on mds daemons: {''mds.exported'', ''mds.imported''}' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-18T20:39:50.543 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T20:39:50.629 INFO:teuthology.run:FAIL